Oct 31 00:27:15.630081 ip-10-0-143-105 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:27:15.984794 ip-10-0-143-105 kubenswrapper[2545]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:27:15.984794 ip-10-0-143-105 kubenswrapper[2545]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:27:15.984794 ip-10-0-143-105 kubenswrapper[2545]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:27:15.984794 ip-10-0-143-105 kubenswrapper[2545]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:27:15.989637 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989541 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:15.989637 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989633 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:15.989637 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989637 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:15.989637 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989641 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:15.989637 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989645 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:15.989637 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989648 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989651 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989654 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989657 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989660 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989663 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989665 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989668 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989670 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989673 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989676 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989678 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989682 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989685 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989688 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989690 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989693 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989696 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989700 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989704 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:15.989881 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989707 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989710 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989713 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989715 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989718 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989721 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989729 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989733 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989735 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989738 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989740 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989743 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989746 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989748 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989752 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989754 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989758 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989761 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989763 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989766 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:15.990350 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989768 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989771 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989773 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989776 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989779 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989796 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989799 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989803 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989805 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989809 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989811 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989814 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989817 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989820 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989823 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989825 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989828 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989831 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989836 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989838 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:15.990830 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989841 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989844 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989847 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989850 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989853 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989855 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989860 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989865 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989868 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989871 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989873 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989876 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989879 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989882 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989884 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989887 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989890 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989893 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989897 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:15.991296 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989899 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989902 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989905 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989907 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989910 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989913 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989915 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989918 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989921 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989924 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989926 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989929 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989931 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989934 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989938 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989940 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989943 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989947 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989950 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989954 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:15.991763 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989956 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.989959 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990826 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990834 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990837 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990840 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990843 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990846 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990849 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990852 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990855 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990857 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990861 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990863 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990866 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990870 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990873 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990876 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990878 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:15.992273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990881 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990884 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990888 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990892 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990895 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990898 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990901 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990904 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990907 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990910 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990913 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990916 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990919 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990922 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990925 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990928 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990930 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990933 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990936 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:15.992734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990939 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990941 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990944 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990946 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990949 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990952 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990955 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990958 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990961 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990963 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990966 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990968 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990971 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990973 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990976 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990978 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990981 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990984 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990986 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990989 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990992 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:15.993244 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990995 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.990998 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991001 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991004 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991007 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991010 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991013 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991016 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991020 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991023 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991026 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991028 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991031 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991034 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991036 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991039 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991041 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991044 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991047 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:15.993752 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991053 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991056 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991058 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991061 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991063 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991066 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991069 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991071 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991074 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991076 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991079 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991082 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991085 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991087 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991090 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991093 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991096 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991099 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991102 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991104 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:15.994225 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991106 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991109 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991112 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991114 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991117 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991120 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991123 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991126 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991129 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991131 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991611 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991617 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991622 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991627 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991630 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991633 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991636 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991639 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991642 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:15.994734 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991644 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991647 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991650 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991653 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991656 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991659 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991662 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991664 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991669 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991673 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991676 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991679 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991682 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991685 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991687 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991690 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991693 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991695 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991698 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:15.995215 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991701 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991703 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991706 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991708 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991711 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991713 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991716 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991719 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991722 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991726 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991729 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991732 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991734 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991737 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991739 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991742 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991745 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991747 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991750 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:15.995666 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991752 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991755 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991758 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991761 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991764 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991766 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991770 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991772 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991775 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991778 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991793 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991797 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991799 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991802 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991804 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991807 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991810 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991812 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991815 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991818 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:15.996147 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991820 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991823 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991827 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991829 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991832 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991835 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991837 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991847 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991850 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991853 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991855 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991858 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991860 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991863 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991866 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991869 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991871 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991874 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991877 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:15.996617 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991880 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991882 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991885 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991887 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991890 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991893 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991895 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991898 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991901 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991904 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991906 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991910 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991913 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991915 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991918 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991921 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991925 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991927 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991930 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.991932 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:15.997124 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992722 2545 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992733 2545 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992741 2545 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992747 2545 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992751 2545 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992755 2545 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992759 2545 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992763 2545 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992766 2545 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992769 2545 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992773 2545 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992776 2545 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992780 2545 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992795 2545 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992799 2545 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992802 2545 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992805 2545 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992808 2545 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992812 2545 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992815 2545 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992818 2545 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992822 2545 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992825 2545 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992829 2545 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:27:15.997615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992834 2545 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992837 2545 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992841 2545 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992844 2545 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992848 2545 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992851 2545 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992854 2545 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992857 2545 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992861 2545 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992864 2545 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992867 2545 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992871 2545 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992874 2545 flags.go:64] FLAG: --enable-server="true" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992877 2545 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992882 2545 flags.go:64] FLAG: --event-burst="100" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992885 2545 flags.go:64] FLAG: --event-qps="50" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992888 2545 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992891 2545 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992895 2545 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992899 2545 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992903 2545 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992906 2545 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992909 2545 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992912 2545 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992915 2545 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:27:15.998186 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992918 2545 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992921 2545 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992923 2545 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992926 2545 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992929 2545 flags.go:64] FLAG: --feature-gates="" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992933 2545 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992936 2545 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992940 2545 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992943 2545 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992946 2545 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992950 2545 flags.go:64] FLAG: --help="false" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992953 2545 flags.go:64] FLAG: --hostname-override="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992957 2545 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992960 2545 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992963 2545 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992967 2545 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992970 2545 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992973 2545 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992976 2545 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992979 2545 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992983 2545 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992986 2545 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992989 2545 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992991 2545 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:27:15.998900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992995 2545 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.992998 2545 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993001 2545 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993004 2545 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993006 2545 flags.go:64] FLAG: --lock-file="" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993009 2545 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993012 2545 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993015 2545 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993021 2545 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993024 2545 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993027 2545 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993030 2545 flags.go:64] FLAG: --logging-format="text" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993032 2545 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993037 2545 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993040 2545 flags.go:64] FLAG: --manifest-url="" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993044 2545 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993049 2545 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993052 2545 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993056 2545 flags.go:64] FLAG: --max-pods="110" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993059 2545 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993062 2545 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993065 2545 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993068 2545 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993071 2545 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993074 2545 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:27:15.999495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993077 2545 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993084 2545 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993088 2545 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993091 2545 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993094 2545 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993097 2545 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993100 2545 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993103 2545 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993106 2545 flags.go:64] FLAG: --port="10250" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993109 2545 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993112 2545 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-027a027a8c2c19887" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993115 2545 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993118 2545 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993121 2545 flags.go:64] FLAG: --register-node="true" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993124 2545 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993128 2545 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993131 2545 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993133 2545 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993136 2545 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993140 2545 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993144 2545 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993147 2545 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993152 2545 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993155 2545 flags.go:64] FLAG: --runonce="false" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993158 2545 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993161 2545 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993164 2545 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:27:16.000116 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993167 2545 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993170 2545 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993174 2545 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993177 2545 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993180 2545 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993183 2545 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993186 2545 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993189 2545 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993192 2545 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993195 2545 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993198 2545 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993201 2545 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993206 2545 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993209 2545 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993212 2545 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993216 2545 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993219 2545 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993221 2545 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993224 2545 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993227 2545 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993230 2545 flags.go:64] FLAG: --v="2" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993235 2545 flags.go:64] FLAG: --version="false" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993239 2545 flags.go:64] FLAG: --vmodule="" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993243 2545 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993247 2545 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:27:16.000769 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993402 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993407 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993411 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993415 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993420 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993423 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993426 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993429 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993432 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993436 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993439 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993442 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993445 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993447 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993450 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993452 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993455 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993458 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993460 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:16.001377 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993463 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993466 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993468 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993471 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993474 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993476 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993479 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993482 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993485 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993487 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993490 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993492 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993495 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993499 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993502 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993506 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993508 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993511 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993514 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993517 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:16.001892 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993519 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993522 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993524 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993527 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993530 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993533 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993535 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993538 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993541 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993543 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993546 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993549 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993551 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993554 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993556 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993559 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993562 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993564 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993567 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:16.002398 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993570 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993572 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993574 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993577 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993579 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993582 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993586 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993589 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993593 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993595 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993598 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993601 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993603 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993606 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993609 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993613 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993616 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993618 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993622 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:16.002888 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993624 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993627 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993629 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993632 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993635 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993637 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993640 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993643 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993646 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993648 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993651 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993653 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993656 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993659 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993662 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993664 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993666 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993669 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993672 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993675 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:16.003346 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993678 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993682 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993684 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993687 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993690 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993692 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993695 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993698 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993700 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:15.993727 2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993763 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993767 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993771 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993775 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993777 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993791 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:16.003859 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993795 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993797 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993800 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993803 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993806 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993808 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993811 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993814 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993816 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993819 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993822 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993824 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993828 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993832 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993835 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993837 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993841 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993845 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993848 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:16.004251 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993850 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993853 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993856 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993859 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993862 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993864 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993867 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993869 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993872 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993875 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993878 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993880 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993883 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993886 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993888 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993891 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993893 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993896 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993898 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993901 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:16.004694 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993904 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993907 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993909 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993912 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993914 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993917 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993920 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993923 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993926 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993930 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993935 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993939 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993942 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993945 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993947 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993950 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993953 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993956 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993959 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:16.005185 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993961 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993964 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993966 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993969 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993972 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993974 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993977 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993980 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993982 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993985 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993987 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993990 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993993 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993995 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.993998 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994000 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994004 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994007 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994010 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994012 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:16.005633 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994015 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994017 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994021 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994026 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994028 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994031 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994034 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994036 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994039 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994041 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994044 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994047 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994049 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994052 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994055 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994058 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994061 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994064 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994066 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:16.006117 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994069 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994072 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:15.994074 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.000539 2545 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.000553 2545 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.001202 2545 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.001210 2545 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001284 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001290 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001295 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001298 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001302 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001304 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001307 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001310 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001312 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001315 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001317 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001320 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:16.006586 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001323 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001325 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001328 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001331 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001333 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001336 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001339 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001341 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001344 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001346 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001349 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001352 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001356 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001358 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001363 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001367 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001371 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001374 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001377 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:16.007081 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001380 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001383 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001387 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001391 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001395 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001399 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001401 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001404 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001407 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001410 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001412 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001415 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001418 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001421 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001423 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001426 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001429 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001431 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001434 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:16.007539 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001437 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001439 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001442 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001445 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001448 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001451 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001454 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001456 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001459 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001461 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001464 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001467 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001470 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001472 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001475 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001478 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001480 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001483 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001485 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001488 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:16.008005 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001491 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001493 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001496 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001498 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001501 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001504 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001507 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001509 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001512 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001514 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001517 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001520 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001522 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001525 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001527 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001530 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001532 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001535 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001537 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001540 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:16.008478 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001543 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001546 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001549 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001551 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001554 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001556 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001559 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001561 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001564 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001566 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001569 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001571 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001574 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001576 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001579 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001581 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:16.008992 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.001609 2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001745 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001751 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001754 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001758 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001761 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001765 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001767 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001770 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001773 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001775 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001778 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001795 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001798 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001801 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001803 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001807 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001809 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001812 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:16.009355 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001815 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001818 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001820 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001823 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001825 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001828 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001831 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001834 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001836 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001839 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001842 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001846 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001849 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001852 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001855 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001858 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001860 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001863 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001866 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:16.009818 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001869 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001871 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001874 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001877 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001881 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001884 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001887 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001889 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001892 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001895 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001897 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001901 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001904 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001907 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001910 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001912 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001914 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001917 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001920 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001922 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:16.010378 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001925 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001927 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001930 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001932 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001935 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001937 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001940 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001942 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001945 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001947 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001950 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001952 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001955 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001957 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001960 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001962 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001965 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001967 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001970 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001973 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:16.011325 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001975 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001978 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001980 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001982 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001986 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001988 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001991 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001993 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001996 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.001998 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002001 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002003 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002006 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002008 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002010 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002013 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002015 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002018 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002021 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:16.011814 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002023 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002026 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002029 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002031 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002034 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002036 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002039 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002042 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002044 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002047 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.002070 2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002102 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002106 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002109 2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002111 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002114 2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:16.012273 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002116 2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002119 2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002123 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002125 2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002128 2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002130 2545 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002133 2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002136 2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002138 2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002141 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002143 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002145 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002148 2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002151 2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002153 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002156 2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002158 2545 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002164 2545 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002167 2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002169 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:16.012667 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002172 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002175 2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002177 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002180 2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002182 2545 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002185 2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002187 2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002190 2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002192 2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002195 2545 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002197 2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002199 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002202 2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002205 2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002207 2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002210 2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002213 2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002215 2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002218 2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002220 2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:16.013154 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002223 2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002225 2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002228 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002231 2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002233 2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002236 2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002238 2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002241 2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002243 2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002247 2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002250 2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002252 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002256 2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002260 2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002264 2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002267 2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002269 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002272 2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002275 2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:16.013616 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002277 2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002280 2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002282 2545 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002285 2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002287 2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002290 2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002293 2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002298 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002301 2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002304 2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002309 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002312 2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002314 2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002317 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002319 2545 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002322 2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002325 2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002327 2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002330 2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002332 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:16.014080 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002335 2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002337 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002341 2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002344 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002346 2545 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002349 2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002352 2545 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002354 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002357 2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002359 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002362 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002364 2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002367 2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002369 2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002372 2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002374 2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002377 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002379 2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002382 2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002386 2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:16.014540 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002389 2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:16.002391 2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.002441 2545 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.004165 2545 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.004964 2545 server.go:1014] "Starting client certificate rotation" Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.005065 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.005100 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.006624 2545 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:27:16.015148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.012156 2545 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:27:16.021452 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.021435 2545 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:27:16.024222 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.024209 2545 log.go:25] "Validated CRI v1 image API" Oct 31 00:27:16.025055 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.025034 2545 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:27:16.030195 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.030176 2545 fs.go:135] Filesystem UUIDs: map[03dcec63-aa92-47fd-a621-76cd90240bf1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9b27e265-8066-4aa4-816d-8f33486942be:/dev/nvme0n1p4] Oct 31 00:27:16.030261 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.030195 2545 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:27:16.035404 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.035300 2545 manager.go:217] Machine: {Timestamp:2026-10-31 00:27:16.033907057 +0000 UTC m=+0.300237843 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099772 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29f4aee6f889f5dd802221e947e72a SystemUUID:ec29f4ae-e6f8-89f5-dd80-2221e947e72a BootID:546b548d-ec7e-445f-a673-0352fe144aa1 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ff:97:52:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ff:97:52:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:f4:b0:52:91:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:27:16.035404 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.035402 2545 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:27:16.035499 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.035477 2545 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:27:16.038688 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.038667 2545 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:27:16.038833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.038690 2545 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-105.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:27:16.038880 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.038836 2545 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:27:16.038880 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.038844 2545 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:27:16.038880 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.038869 2545 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:27:16.038955 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.038885 2545 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:27:16.039767 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.039756 2545 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:27:16.043395 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.043383 2545 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:27:16.043513 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.043504 2545 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:27:16.044174 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.044164 2545 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:27:16.044210 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.044177 2545 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:27:16.044210 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.044188 2545 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:27:16.044210 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.044197 2545 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:27:16.044210 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.044205 2545 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:27:16.044591 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.044576 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:27:16.046710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.046690 2545 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:27:16.051447 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.051434 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:27:16.052340 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052328 2545 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:27:16.052378 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052358 2545 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:27:16.052528 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052519 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:27:16.052564 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052535 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:27:16.052564 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052541 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:27:16.052564 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052546 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:27:16.052564 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052552 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:27:16.052564 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.052558 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:27:16.053026 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.053017 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:27:16.053060 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.053027 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:27:16.053060 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.053035 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:27:16.053060 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.053040 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:27:16.053060 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.053049 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:27:16.053060 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.053057 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:27:16.054270 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.054260 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:27:16.054270 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.054270 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:27:16.057774 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.057759 2545 server.go:1263] "Started kubelet" Oct 31 00:27:16.057921 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.057895 2545 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:27:16.061017 ip-10-0-143-105 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:27:16.062435 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.060801 2545 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:27:16.062435 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.062257 2545 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:27:16.063405 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.063383 2545 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:27:16.065170 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.065150 2545 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:27:16.068597 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.068580 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:27:16.068755 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.068737 2545 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:27:16.068986 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.068968 2545 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:27:16.069278 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.069260 2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" Oct 31 00:27:16.069465 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.069444 2545 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:27:16.069571 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.069474 2545 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:27:16.069571 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.069515 2545 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:27:16.069674 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.069584 2545 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:27:16.069674 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.069591 2545 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:27:16.070097 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.070083 2545 factory.go:55] Registering systemd factory Oct 31 00:27:16.070170 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.070101 2545 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:27:16.070350 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.070336 2545 factory.go:153] Registering CRI-O factory Oct 31 00:27:16.070350 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.070352 2545 factory.go:223] Registration of the crio container factory successfully Oct 31 00:27:16.070463 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.070388 2545 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:27:16.070463 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.070402 2545 factory.go:103] Registering Raw factory Oct 31 00:27:16.070463 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.070412 2545 manager.go:1199] Started watching for new ooms in manager Oct 31 00:27:16.071170 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.071155 2545 manager.go:319] Starting recovery of all containers Oct 31 00:27:16.082634 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.082611 2545 manager.go:324] Recovery completed Oct 31 00:27:16.089645 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.089629 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:16.091148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.091124 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:16.091222 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.091162 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:16.091222 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.091175 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:16.091681 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.091665 2545 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:27:16.091681 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.091678 2545 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:27:16.091811 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.091694 2545 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:27:16.091811 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.091666 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:16.092445 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.092430 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:16.092492 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.092456 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:16.092492 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.092468 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:16.093750 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.093739 2545 policy_none.go:50] "Start" Oct 31 00:27:16.093796 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.093755 2545 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:27:16.093796 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.093768 2545 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:27:16.095752 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.095736 2545 policy_none.go:44] "Start" Oct 31 00:27:16.141344 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.141314 2545 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:27:16.143666 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.143650 2545 manager.go:341] "Starting Device Plugin manager" Oct 31 00:27:16.143756 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.143687 2545 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:27:16.143756 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.143700 2545 server.go:87] "Starting device plugin registration server" Oct 31 00:27:16.144005 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.143993 2545 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:27:16.144047 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.144007 2545 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:27:16.144115 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.144093 2545 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:27:16.144204 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.144192 2545 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:27:16.144256 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.144206 2545 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:27:16.144839 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.144822 2545 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:27:16.144926 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.144881 2545 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" Oct 31 00:27:16.206675 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.206640 2545 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-105.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:27:16.206675 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.206633 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:27:16.206890 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.206653 2545 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-105.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:27:16.207942 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.207924 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:27:16.208701 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.208684 2545 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:27:16.208812 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.208729 2545 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:27:16.208812 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.208772 2545 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:27:16.244694 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.244621 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:16.245615 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.245598 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:16.245726 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.245632 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:16.245726 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.245646 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:16.245726 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.245679 2545 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.256677 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.255890 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-105.us-west-2.compute.internal.18e375a6d71ed6f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-105.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-105.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:27:16.057732851 +0000 UTC m=+0.324063637,LastTimestamp:2026-10-31 00:27:16.057732851 +0000 UTC m=+0.324063637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-105.us-west-2.compute.internal,}" Oct 31 00:27:16.280001 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.279970 2545 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:27:16.280123 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.280050 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mxghq" Oct 31 00:27:16.309097 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.309067 2545 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal"] Oct 31 00:27:16.309223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.309141 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:16.310139 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.310120 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:16.310241 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.310154 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:16.310241 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.310169 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:16.312377 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.312361 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:16.312509 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.312494 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.312559 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.312532 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:16.313719 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.313699 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:16.313809 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.313732 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:16.313809 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.313743 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:16.313809 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.313707 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:16.313951 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.313828 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:16.313951 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.313845 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:16.316207 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.316189 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.316284 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.316216 2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:16.317016 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.317003 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:16.317106 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.317026 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:16.317106 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.317036 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:16.341374 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.341151 2545 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" node="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.344044 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.344022 2545 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" node="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.359073 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.359054 2545 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:16.360375 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.360361 2545 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:16.371438 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.371418 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.371508 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.371443 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d22eb601a1d03eddfb99de6ecc9e9dbd-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"d22eb601a1d03eddfb99de6ecc9e9dbd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.371508 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.371460 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.451618 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.451592 2545 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:16.452148 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.452131 2545 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:16.453091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.453074 2545 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.453152 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:16.453098 2545 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-105.us-west-2.compute.internal\": node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" Oct 31 00:27:16.469582 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.469562 2545 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.471701 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.471682 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.471773 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.471715 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.471773 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.471757 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d22eb601a1d03eddfb99de6ecc9e9dbd-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"d22eb601a1d03eddfb99de6ecc9e9dbd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.472081 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.471812 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.472367 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.472163 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.472367 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.472199 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d22eb601a1d03eddfb99de6ecc9e9dbd-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"d22eb601a1d03eddfb99de6ecc9e9dbd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.580337 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.580256 2545 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:27:16.581104 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.581088 2545 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.644494 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.644468 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.646625 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.646606 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:27:16.687182 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:16.687156 2545 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:27:17.005386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.005310 2545 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:27:17.005868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.005489 2545 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:27:17.005868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.005489 2545 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:27:17.005868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.005509 2545 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:27:17.005868 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.005466 2545 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a5bf11d8d49064afc9ef2bd5f2126127-faee8f219fa21b80.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-105.us-west-2.compute.internal.18e375a6d91cb264\": read tcp 10.0.143.105:34160->18.190.165.24:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-105.us-west-2.compute.internal.18e375a6d91cb264 default 4880 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-105.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-105.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-105.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:27:16 +0000 UTC,LastTimestamp:2026-10-31 00:27:16.310135656 +0000 UTC m=+0.576466442,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-105.us-west-2.compute.internal,}" Oct 31 00:27:17.045276 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.045250 2545 apiserver.go:52] "Watching apiserver" Oct 31 00:27:17.069103 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.069080 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:27:17.284806 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.284711 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:22:16 +0000 UTC" deadline="2028-06-27 20:09:26.24209246 +0000 UTC" Oct 31 00:27:17.284806 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.284741 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14539h42m8.957354644s" Oct 31 00:27:17.313899 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.313879 2545 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:27:17.314228 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.314209 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-nt477","kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-ksxrk","openshift-dns/node-resolver-946fm","openshift-network-diagnostics/network-check-target-8grlc","openshift-network-operator/iptables-alerter-qlblw","openshift-ovn-kubernetes/ovnkube-node-c2rj2","openshift-cluster-node-tuning-operator/tuned-mdpsp","openshift-image-registry/node-ca-fgvdh","openshift-multus/multus-additional-cni-plugins-zz7n4","openshift-multus/multus-nn2wc","openshift-multus/network-metrics-daemon-mf88n"] Oct 31 00:27:17.316889 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.316865 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.319828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.319807 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.322101 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.322084 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.322197 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.322155 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:17.324352 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.324334 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.326423 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.326410 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:17.326479 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.326459 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:17.328740 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.328720 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.331188 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.331172 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.333535 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.333517 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.335975 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.335956 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.338479 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.338460 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.341090 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.341070 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.343595 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.343579 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:17.343656 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.343641 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:17.369891 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.369847 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:27:17.369996 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.369947 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:27:17.369996 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.369975 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-78fsr\"" Oct 31 00:27:17.373256 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.371993 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rlgrr\"" Oct 31 00:27:17.373256 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.372410 2545 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:27:17.373256 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.372423 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:27:17.373256 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.372435 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:27:17.373256 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.372753 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:27:17.376140 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376118 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-registration-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.376232 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376154 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.376232 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376175 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-netns\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376232 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376191 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376232 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376207 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.376232 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376222 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-systemd\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376276 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-slash\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376310 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-conf\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376327 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376366 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376398 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f48r\" (UniqueName: \"kubernetes.io/projected/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-kube-api-access-9f48r\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376419 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-netd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376443 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwx2\" (UniqueName: \"kubernetes.io/projected/79c0a7af-0b6e-4056-8e2d-166885cb98dc-kube-api-access-pkwx2\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.376495 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376478 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-socket-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376504 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj77k\" (UniqueName: \"kubernetes.io/projected/fee32755-d814-417f-8311-6a1dc5b60fa5-kube-api-access-rj77k\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376545 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-systemd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376561 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376587 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-etc-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376623 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376650 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-log-socket\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376671 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-modprobe-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376678 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376687 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-run\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376718 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7dm\" (UniqueName: \"kubernetes.io/projected/241ccb66-4fd3-427c-9b7f-279439013bfd-kube-api-access-wf7dm\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376747 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376757 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rw8bx\"" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376772 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-sys-fs\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376820 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-var-lib-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376861 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:17.376916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376893 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf86f\" (UniqueName: \"kubernetes.io/projected/d172f407-b9ac-4193-b0f4-fab1c53e6874-kube-api-access-tf86f\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376929 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376952 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-k8s-cni-cncf-io\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376976 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtjc\" (UniqueName: \"kubernetes.io/projected/9316842b-0133-456e-abeb-e0b38e7f04ff-kube-api-access-hjtjc\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.376992 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63af1f1-ad79-4694-955a-25eb509899d7-agent-certs\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377016 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377031 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-bin\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377070 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-netns\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377096 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af99842a-26b2-4e57-aae5-3aceef8f0776-hosts-file\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377117 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-conf-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377134 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8057563-4fad-49a2-8d82-4902eca5fa57-host-slash\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377200 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377234 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-tuned\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377249 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d172f407-b9ac-4193-b0f4-fab1c53e6874-serviceca\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377263 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-cni-binary-copy\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377277 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-bin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.377710 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377295 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2h76\" (UniqueName: \"kubernetes.io/projected/c8057563-4fad-49a2-8d82-4902eca5fa57-kube-api-access-g2h76\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377312 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377327 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-tmp\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377352 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-kubelet\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377369 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8057563-4fad-49a2-8d82-4902eca5fa57-iptables-alerter-script\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377383 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-script-lib\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377397 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-kubernetes\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377418 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-lib-modules\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377435 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cnibin\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377455 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-daemon-config\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377480 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-multus-certs\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377520 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-etc-kubernetes\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377544 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377574 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-dbus\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377608 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zw2\" (UniqueName: \"kubernetes.io/projected/af99842a-26b2-4e57-aae5-3aceef8f0776-kube-api-access-54zw2\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377635 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-ovn\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.378562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377656 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-node-log\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377672 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-config\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377702 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovn-node-metrics-cert\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377724 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377741 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-sys\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377756 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d172f407-b9ac-4193-b0f4-fab1c53e6874-host\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377782 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-system-cni-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377861 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-system-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377887 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-multus\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377917 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63af1f1-ad79-4694-955a-25eb509899d7-konnectivity-ca\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377947 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-env-overrides\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377973 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwkw\" (UniqueName: \"kubernetes.io/projected/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-kube-api-access-jnwkw\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.377998 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-device-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378038 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-kubelet-config\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378057 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99842a-26b2-4e57-aae5-3aceef8f0776-tmp-dir\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378071 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-kubelet\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.379386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378087 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-systemd-units\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.380223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378110 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysconfig\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.380223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378136 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-var-lib-kubelet\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.380223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378151 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-host\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.380223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378175 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-cnibin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.380223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378198 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-socket-dir-parent\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.380223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.378235 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-hostroot\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.381040 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.381019 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:17.381040 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.381033 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:17.381126 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.381112 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:27:17.381182 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.381167 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jsqnl\"" Oct 31 00:27:17.383309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.383284 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:27:17.383638 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.383615 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:27:17.383677 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.383641 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:27:17.383677 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.383650 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:27:17.383749 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.383738 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:27:17.419362 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.419338 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lwt4s\"" Oct 31 00:27:17.419362 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.419341 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-46b97\"" Oct 31 00:27:17.419772 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.419755 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:27:17.419828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.419779 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:27:17.419828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.419791 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:17.422032 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.422015 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6gdh\"" Oct 31 00:27:17.422098 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.422076 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:27:17.422098 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.422080 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:27:17.422098 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.422085 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2znq\"" Oct 31 00:27:17.424384 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.424367 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:17.426845 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.426832 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:27:17.426917 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.426892 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:27:17.431138 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.431118 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:27:17.431229 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.431173 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:27:17.431291 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.431235 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cdw8b\"" Oct 31 00:27:17.431291 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.431239 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:27:17.431291 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.431271 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:27:17.431404 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.431316 2545 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:27:17.431404 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.431369 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cwnvz" Oct 31 00:27:17.479149 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479124 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-sys\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479149 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479150 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d172f407-b9ac-4193-b0f4-fab1c53e6874-host\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479167 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-system-cni-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479183 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-system-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479200 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-multus\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479215 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63af1f1-ad79-4694-955a-25eb509899d7-konnectivity-ca\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479232 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-env-overrides\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479256 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d172f407-b9ac-4193-b0f4-fab1c53e6874-host\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479275 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-multus\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479296 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-sys\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479304 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwkw\" (UniqueName: \"kubernetes.io/projected/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-kube-api-access-jnwkw\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.479309 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479257 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-system-cni-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479281 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-system-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479483 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-device-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479518 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-kubelet-config\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479546 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99842a-26b2-4e57-aae5-3aceef8f0776-tmp-dir\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479572 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-kubelet\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479569 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-device-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479600 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-systemd-units\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479592 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-kubelet-config\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479641 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysconfig\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479663 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-systemd-units\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479672 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-var-lib-kubelet\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479685 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysconfig\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479642 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-kubelet\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479707 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-var-lib-kubelet\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479701 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-host\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479766 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-host\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479795 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-env-overrides\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.479819 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479820 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-cnibin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479819 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63af1f1-ad79-4694-955a-25eb509899d7-konnectivity-ca\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479906 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-cnibin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479911 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99842a-26b2-4e57-aae5-3aceef8f0776-tmp-dir\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479914 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-socket-dir-parent\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479951 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-hostroot\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479978 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-registration-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.479998 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-hostroot\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480005 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480025 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-socket-dir-parent\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480034 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-netns\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480049 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480028 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-registration-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480063 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480080 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480085 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-netns\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480082 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480114 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-systemd\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.480723 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480129 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-slash\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480146 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-conf\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480163 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480179 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-slash\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480185 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480209 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-systemd\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480211 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f48r\" (UniqueName: \"kubernetes.io/projected/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-kube-api-access-9f48r\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480253 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-netd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480264 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480277 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480282 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwx2\" (UniqueName: \"kubernetes.io/projected/79c0a7af-0b6e-4056-8e2d-166885cb98dc-kube-api-access-pkwx2\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480300 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-netd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480312 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-conf\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480326 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-socket-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480348 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj77k\" (UniqueName: \"kubernetes.io/projected/fee32755-d814-417f-8311-6a1dc5b60fa5-kube-api-access-rj77k\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480363 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-systemd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480379 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-etc-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.481440 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480412 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-systemd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480417 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-etc-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480416 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-socket-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480440 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480463 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-log-socket\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480487 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-modprobe-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480502 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-run\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480511 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-log-socket\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480517 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7dm\" (UniqueName: \"kubernetes.io/projected/241ccb66-4fd3-427c-9b7f-279439013bfd-kube-api-access-wf7dm\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480535 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480550 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-sys-fs\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480554 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-run\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480572 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-var-lib-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.480585 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480601 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480609 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-sys-fs\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480610 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-modprobe-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480626 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf86f\" (UniqueName: \"kubernetes.io/projected/d172f407-b9ac-4193-b0f4-fab1c53e6874-kube-api-access-tf86f\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.482221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480634 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-var-lib-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.480661 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs podName:77f7bca8-ea23-4b2b-9eaa-b26425121ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:17.980640522 +0000 UTC m=+2.246971310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs") pod "network-metrics-daemon-mf88n" (UID: "77f7bca8-ea23-4b2b-9eaa-b26425121ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480766 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480776 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480821 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-k8s-cni-cncf-io\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480856 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtjc\" (UniqueName: \"kubernetes.io/projected/9316842b-0133-456e-abeb-e0b38e7f04ff-kube-api-access-hjtjc\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480881 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63af1f1-ad79-4694-955a-25eb509899d7-agent-certs\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480886 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-k8s-cni-cncf-io\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480910 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480938 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-bin\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480965 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-netns\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.480991 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af99842a-26b2-4e57-aae5-3aceef8f0776-hosts-file\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481014 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-conf-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481019 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-bin\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481033 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8057563-4fad-49a2-8d82-4902eca5fa57-host-slash\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481039 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-netns\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481051 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.483077 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481056 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-conf-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.481015 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481086 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481088 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af99842a-26b2-4e57-aae5-3aceef8f0776-hosts-file\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481105 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8057563-4fad-49a2-8d82-4902eca5fa57-host-slash\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481067 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-tuned\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.481115 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret podName:09934062-de9f-4e88-b25e-4f4a7947e4ce nodeName:}" failed. No retries permitted until 2026-10-31 00:27:17.981096999 +0000 UTC m=+2.247427777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret") pod "global-pull-secret-syncer-ksxrk" (UID: "09934062-de9f-4e88-b25e-4f4a7947e4ce") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481159 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d172f407-b9ac-4193-b0f4-fab1c53e6874-serviceca\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481178 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-cni-binary-copy\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481193 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-bin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481219 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2h76\" (UniqueName: \"kubernetes.io/projected/c8057563-4fad-49a2-8d82-4902eca5fa57-kube-api-access-g2h76\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481231 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-bin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481246 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481284 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-tmp\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481299 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-kubelet\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481313 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8057563-4fad-49a2-8d82-4902eca5fa57-iptables-alerter-script\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481328 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-script-lib\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.483964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481343 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-kubernetes\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481361 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-lib-modules\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481388 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cnibin\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481403 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-daemon-config\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481419 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-multus-certs\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481423 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481444 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-etc-kubernetes\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481453 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481474 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481488 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-kubernetes\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481490 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-dbus\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481527 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zw2\" (UniqueName: \"kubernetes.io/projected/af99842a-26b2-4e57-aae5-3aceef8f0776-kube-api-access-54zw2\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481549 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-dbus\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481557 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-ovn\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481634 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-lib-modules\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481660 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-node-log\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481663 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-multus-certs\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481681 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-etc-kubernetes\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.484828 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481669 2545 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481701 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-config\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481707 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-ovn\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481731 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481739 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovn-node-metrics-cert\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481768 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481781 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cnibin\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481782 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481805 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-kubelet\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481884 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-node-log\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.481898 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d172f407-b9ac-4193-b0f4-fab1c53e6874-serviceca\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.482043 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.482328 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-daemon-config\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.482649 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-cni-binary-copy\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.483445 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8057563-4fad-49a2-8d82-4902eca5fa57-iptables-alerter-script\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.484216 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-script-lib\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.484251 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-config\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.485612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.485312 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-tuned\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.486134 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.485462 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-tmp\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.486134 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.485516 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63af1f1-ad79-4694-955a-25eb509899d7-agent-certs\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.486134 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.486114 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovn-node-metrics-cert\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.543552 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.543487 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f48r\" (UniqueName: \"kubernetes.io/projected/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-kube-api-access-9f48r\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:17.543936 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.543909 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwkw\" (UniqueName: \"kubernetes.io/projected/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-kube-api-access-jnwkw\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.544022 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.543905 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwx2\" (UniqueName: \"kubernetes.io/projected/79c0a7af-0b6e-4056-8e2d-166885cb98dc-kube-api-access-pkwx2\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.550614 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.550582 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2h76\" (UniqueName: \"kubernetes.io/projected/c8057563-4fad-49a2-8d82-4902eca5fa57-kube-api-access-g2h76\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.550700 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.550612 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zw2\" (UniqueName: \"kubernetes.io/projected/af99842a-26b2-4e57-aae5-3aceef8f0776-kube-api-access-54zw2\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.555635 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.555607 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj77k\" (UniqueName: \"kubernetes.io/projected/fee32755-d814-417f-8311-6a1dc5b60fa5-kube-api-access-rj77k\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.557468 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.557442 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7dm\" (UniqueName: \"kubernetes.io/projected/241ccb66-4fd3-427c-9b7f-279439013bfd-kube-api-access-wf7dm\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.557945 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.557922 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtjc\" (UniqueName: \"kubernetes.io/projected/9316842b-0133-456e-abeb-e0b38e7f04ff-kube-api-access-hjtjc\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.559346 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.559329 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:17.559409 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.559348 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:17.559409 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.559358 2545 projected.go:196] Error preparing data for projected volume kube-api-access-t8q8z for pod openshift-network-diagnostics/network-check-target-8grlc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:17.559477 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.559437 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z podName:a33cb5d1-e145-47db-9c98-2561a4213883 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:18.059421925 +0000 UTC m=+2.325752716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t8q8z" (UniqueName: "kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z") pod "network-check-target-8grlc" (UID: "a33cb5d1-e145-47db-9c98-2561a4213883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:17.564068 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.564048 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf86f\" (UniqueName: \"kubernetes.io/projected/d172f407-b9ac-4193-b0f4-fab1c53e6874-kube-api-access-tf86f\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.658157 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.658125 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:17.680960 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.680937 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:27:17.683243 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.683224 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-946fm" Oct 31 00:27:17.698686 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.698662 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:27:17.731241 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.731216 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:17.733738 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.733721 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:27:17.745111 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.745087 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:27:17.776131 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.776111 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nn2wc" Oct 31 00:27:17.780971 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.780949 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:27:17.893374 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.893306 2545 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:17.948248 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:17.948208 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63412fdebab41a737ab007fe557fdbf2.slice/crio-5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f WatchSource:0}: Error finding container 5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f: Status 404 returned error can't find the container with id 5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f Oct 31 00:27:17.955310 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.955293 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:27:17.958745 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:17.958722 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22eb601a1d03eddfb99de6ecc9e9dbd.slice/crio-34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b WatchSource:0}: Error finding container 34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b: Status 404 returned error can't find the container with id 34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b Oct 31 00:27:17.986413 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.986382 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:17.986518 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:17.986432 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:17.986571 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.986541 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:17.986616 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.986605 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs podName:77f7bca8-ea23-4b2b-9eaa-b26425121ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:18.986582348 +0000 UTC m=+3.252913127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs") pod "network-metrics-daemon-mf88n" (UID: "77f7bca8-ea23-4b2b-9eaa-b26425121ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:17.986668 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.986541 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:17.986668 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:17.986649 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret podName:09934062-de9f-4e88-b25e-4f4a7947e4ce nodeName:}" failed. No retries permitted until 2026-10-31 00:27:18.986639758 +0000 UTC m=+3.252970533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret") pod "global-pull-secret-syncer-ksxrk" (UID: "09934062-de9f-4e88-b25e-4f4a7947e4ce") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:18.003940 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.003914 2545 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:18.087154 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.087112 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:18.087689 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.087280 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:18.087689 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.087300 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:18.087689 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.087312 2545 projected.go:196] Error preparing data for projected volume kube-api-access-t8q8z for pod openshift-network-diagnostics/network-check-target-8grlc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:18.087689 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.087359 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z podName:a33cb5d1-e145-47db-9c98-2561a4213883 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:19.087344562 +0000 UTC m=+3.353675356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8q8z" (UniqueName: "kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z") pod "network-check-target-8grlc" (UID: "a33cb5d1-e145-47db-9c98-2561a4213883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:18.212975 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.212873 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"d22eb601a1d03eddfb99de6ecc9e9dbd","Type":"ContainerStarted","Data":"34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b"} Oct 31 00:27:18.213900 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.213878 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"63412fdebab41a737ab007fe557fdbf2","Type":"ContainerStarted","Data":"5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f"} Oct 31 00:27:18.237642 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.237617 2545 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:18.435583 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.435526 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:22:17 +0000 UTC" deadline="2028-04-28 05:17:50.836299741 +0000 UTC" Oct 31 00:27:18.435583 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.435573 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13084h50m32.400729098s" Oct 31 00:27:18.679412 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.679377 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee32755_d814_417f_8311_6a1dc5b60fa5.slice/crio-8607ad7be1f639a72522fe3256f13461fc5e4c23e3f1a660e91809b30f61ce9c WatchSource:0}: Error finding container 8607ad7be1f639a72522fe3256f13461fc5e4c23e3f1a660e91809b30f61ce9c: Status 404 returned error can't find the container with id 8607ad7be1f639a72522fe3256f13461fc5e4c23e3f1a660e91809b30f61ce9c Oct 31 00:27:18.687354 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.687327 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8057563_4fad_49a2_8d82_4902eca5fa57.slice/crio-787eb665916078990674eb157d806f8a99ddbaf0b5ab0ac68d6556b7a25d91a6 WatchSource:0}: Error finding container 787eb665916078990674eb157d806f8a99ddbaf0b5ab0ac68d6556b7a25d91a6: Status 404 returned error can't find the container with id 787eb665916078990674eb157d806f8a99ddbaf0b5ab0ac68d6556b7a25d91a6 Oct 31 00:27:18.687906 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.687886 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63af1f1_ad79_4694_955a_25eb509899d7.slice/crio-455f84d7c40eb10d34fac51cc723697c75c82cc6aee4d565cd06aedb57d679bd WatchSource:0}: Error finding container 455f84d7c40eb10d34fac51cc723697c75c82cc6aee4d565cd06aedb57d679bd: Status 404 returned error can't find the container with id 455f84d7c40eb10d34fac51cc723697c75c82cc6aee4d565cd06aedb57d679bd Oct 31 00:27:18.750115 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.750085 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf99842a_26b2_4e57_aae5_3aceef8f0776.slice/crio-644ef22a9afc953fb3f4704023c5a386721cfee1fe596f82893e04f254f75168 WatchSource:0}: Error finding container 644ef22a9afc953fb3f4704023c5a386721cfee1fe596f82893e04f254f75168: Status 404 returned error can't find the container with id 644ef22a9afc953fb3f4704023c5a386721cfee1fe596f82893e04f254f75168 Oct 31 00:27:18.753517 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.753494 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9316842b_0133_456e_abeb_e0b38e7f04ff.slice/crio-5f9fbd2fafd64df2cfe50c61ce867568b5fd152e6b73eaef39ff959c3bca5fc8 WatchSource:0}: Error finding container 5f9fbd2fafd64df2cfe50c61ce867568b5fd152e6b73eaef39ff959c3bca5fc8: Status 404 returned error can't find the container with id 5f9fbd2fafd64df2cfe50c61ce867568b5fd152e6b73eaef39ff959c3bca5fc8 Oct 31 00:27:18.754151 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.754127 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799b2dd6_6d05_4ca1_bc09_79bd1634fbaa.slice/crio-fa58d08bd48ddd010c2eac4a2ec4af8f47902b4cc1186e99823eef9be713fd96 WatchSource:0}: Error finding container fa58d08bd48ddd010c2eac4a2ec4af8f47902b4cc1186e99823eef9be713fd96: Status 404 returned error can't find the container with id fa58d08bd48ddd010c2eac4a2ec4af8f47902b4cc1186e99823eef9be713fd96 Oct 31 00:27:18.772558 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.772530 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241ccb66_4fd3_427c_9b7f_279439013bfd.slice/crio-e8a20efeab363b872497c3dc4dbef0fe38a8e2041b4f5afada2e0c3180d6443a WatchSource:0}: Error finding container e8a20efeab363b872497c3dc4dbef0fe38a8e2041b4f5afada2e0c3180d6443a: Status 404 returned error can't find the container with id e8a20efeab363b872497c3dc4dbef0fe38a8e2041b4f5afada2e0c3180d6443a Oct 31 00:27:18.843361 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.843319 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd172f407_b9ac_4193_b0f4_fab1c53e6874.slice/crio-e1fb504d1880f2e5fdc161ec56d44e5306c094421375ef07005ddd5b036f46ff WatchSource:0}: Error finding container e1fb504d1880f2e5fdc161ec56d44e5306c094421375ef07005ddd5b036f46ff: Status 404 returned error can't find the container with id e1fb504d1880f2e5fdc161ec56d44e5306c094421375ef07005ddd5b036f46ff Oct 31 00:27:18.844392 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:18.844369 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c0a7af_0b6e_4056_8e2d_166885cb98dc.slice/crio-e93f8991e164e93bf181abdcaa43c1414300a76d57dc3bc293eb358881a8b188 WatchSource:0}: Error finding container e93f8991e164e93bf181abdcaa43c1414300a76d57dc3bc293eb358881a8b188: Status 404 returned error can't find the container with id e93f8991e164e93bf181abdcaa43c1414300a76d57dc3bc293eb358881a8b188 Oct 31 00:27:18.993475 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.993402 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:18.993475 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:18.993461 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:18.993646 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.993554 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:18.993646 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.993624 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret podName:09934062-de9f-4e88-b25e-4f4a7947e4ce nodeName:}" failed. No retries permitted until 2026-10-31 00:27:20.993607231 +0000 UTC m=+5.259938005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret") pod "global-pull-secret-syncer-ksxrk" (UID: "09934062-de9f-4e88-b25e-4f4a7947e4ce") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:18.993646 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.993566 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:18.993771 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:18.993665 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs podName:77f7bca8-ea23-4b2b-9eaa-b26425121ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:20.993657155 +0000 UTC m=+5.259987929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs") pod "network-metrics-daemon-mf88n" (UID: "77f7bca8-ea23-4b2b-9eaa-b26425121ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:19.094478 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.094424 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:19.095119 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:19.094610 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:19.095119 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:19.094633 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:19.095119 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:19.094644 2545 projected.go:196] Error preparing data for projected volume kube-api-access-t8q8z for pod openshift-network-diagnostics/network-check-target-8grlc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:19.095119 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:19.094704 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z podName:a33cb5d1-e145-47db-9c98-2561a4213883 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:21.094689114 +0000 UTC m=+5.361019903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8q8z" (UniqueName: "kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z") pod "network-check-target-8grlc" (UID: "a33cb5d1-e145-47db-9c98-2561a4213883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:19.209249 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.209216 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:19.209431 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.209251 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:19.209431 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:19.209339 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:19.209542 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:19.209421 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:19.209542 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.209486 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:19.209671 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:19.209582 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:19.216906 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.216878 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerStarted","Data":"e93f8991e164e93bf181abdcaa43c1414300a76d57dc3bc293eb358881a8b188"} Oct 31 00:27:19.218409 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.218382 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" event={"ID":"241ccb66-4fd3-427c-9b7f-279439013bfd","Type":"ContainerStarted","Data":"e8a20efeab363b872497c3dc4dbef0fe38a8e2041b4f5afada2e0c3180d6443a"} Oct 31 00:27:19.219970 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.219944 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-946fm" event={"ID":"af99842a-26b2-4e57-aae5-3aceef8f0776","Type":"ContainerStarted","Data":"644ef22a9afc953fb3f4704023c5a386721cfee1fe596f82893e04f254f75168"} Oct 31 00:27:19.221499 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.221439 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" event={"ID":"fee32755-d814-417f-8311-6a1dc5b60fa5","Type":"ContainerStarted","Data":"8607ad7be1f639a72522fe3256f13461fc5e4c23e3f1a660e91809b30f61ce9c"} Oct 31 00:27:19.223016 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.222989 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fgvdh" event={"ID":"d172f407-b9ac-4193-b0f4-fab1c53e6874","Type":"ContainerStarted","Data":"e1fb504d1880f2e5fdc161ec56d44e5306c094421375ef07005ddd5b036f46ff"} Oct 31 00:27:19.224414 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.224385 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"fa58d08bd48ddd010c2eac4a2ec4af8f47902b4cc1186e99823eef9be713fd96"} Oct 31 00:27:19.225855 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.225821 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nn2wc" event={"ID":"9316842b-0133-456e-abeb-e0b38e7f04ff","Type":"ContainerStarted","Data":"5f9fbd2fafd64df2cfe50c61ce867568b5fd152e6b73eaef39ff959c3bca5fc8"} Oct 31 00:27:19.227306 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.227278 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nt477" event={"ID":"c63af1f1-ad79-4694-955a-25eb509899d7","Type":"ContainerStarted","Data":"455f84d7c40eb10d34fac51cc723697c75c82cc6aee4d565cd06aedb57d679bd"} Oct 31 00:27:19.229063 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.229034 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlblw" event={"ID":"c8057563-4fad-49a2-8d82-4902eca5fa57","Type":"ContainerStarted","Data":"787eb665916078990674eb157d806f8a99ddbaf0b5ab0ac68d6556b7a25d91a6"} Oct 31 00:27:19.436312 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.436213 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:22:17 +0000 UTC" deadline="2028-03-30 17:43:12.162505129 +0000 UTC" Oct 31 00:27:19.436312 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:19.436251 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12401h15m52.726259492s" Oct 31 00:27:21.011281 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:21.011133 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:21.011281 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:21.011207 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:21.011913 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.011312 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:21.011913 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.011334 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:21.011913 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.011386 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs podName:77f7bca8-ea23-4b2b-9eaa-b26425121ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:25.011365927 +0000 UTC m=+9.277696721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs") pod "network-metrics-daemon-mf88n" (UID: "77f7bca8-ea23-4b2b-9eaa-b26425121ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:21.011913 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.011407 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret podName:09934062-de9f-4e88-b25e-4f4a7947e4ce nodeName:}" failed. No retries permitted until 2026-10-31 00:27:25.011397388 +0000 UTC m=+9.277728180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret") pod "global-pull-secret-syncer-ksxrk" (UID: "09934062-de9f-4e88-b25e-4f4a7947e4ce") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:21.112107 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:21.111718 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:21.112107 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.111969 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:21.112107 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.111991 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:21.112107 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.112005 2545 projected.go:196] Error preparing data for projected volume kube-api-access-t8q8z for pod openshift-network-diagnostics/network-check-target-8grlc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:21.112107 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.112065 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z podName:a33cb5d1-e145-47db-9c98-2561a4213883 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:25.112046555 +0000 UTC m=+9.378377349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8q8z" (UniqueName: "kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z") pod "network-check-target-8grlc" (UID: "a33cb5d1-e145-47db-9c98-2561a4213883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:21.209513 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:21.209478 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:21.209660 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.209629 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:21.209743 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:21.209728 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:21.209854 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.209816 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:21.209923 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:21.209910 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:21.210006 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:21.209989 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:22.243797 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:22.242526 2545 generic.go:356] "Generic (PLEG): container finished" podID="63412fdebab41a737ab007fe557fdbf2" containerID="ea1ab6c01ff87eaad01c49568b716dc5cf28ad59da6a7b820856a48ce5730b6f" exitCode=0 Oct 31 00:27:22.243797 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:22.242792 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"63412fdebab41a737ab007fe557fdbf2","Type":"ContainerDied","Data":"ea1ab6c01ff87eaad01c49568b716dc5cf28ad59da6a7b820856a48ce5730b6f"} Oct 31 00:27:23.209213 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.209173 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:23.209382 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:23.209322 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:23.209456 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.209388 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:23.209507 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:23.209468 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:23.209561 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.209511 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:23.209615 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:23.209576 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:23.250794 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.250047 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"d22eb601a1d03eddfb99de6ecc9e9dbd","Type":"ContainerStarted","Data":"bdc95e6f9a1357e180acd46be9a61cf74ca6d28325b7c40b47507eb99ae091df"} Oct 31 00:27:23.255779 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.255588 2545 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="38546f83033367389251155f2c498cfd2932f4e02da4817865303265a65346b3" exitCode=0 Oct 31 00:27:23.255779 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.255679 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerDied","Data":"38546f83033367389251155f2c498cfd2932f4e02da4817865303265a65346b3"} Oct 31 00:27:23.267952 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.265284 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"63412fdebab41a737ab007fe557fdbf2","Type":"ContainerStarted","Data":"d942cbe568e12cdb19c5a7155eaad526b995a06c4f9d4ba01420bb6863d78bad"} Oct 31 00:27:23.490987 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:23.490843 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" podStartSLOduration=6.490821608 podStartE2EDuration="6.490821608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:23.366662284 +0000 UTC m=+7.632993082" watchObservedRunningTime="2026-10-31 00:27:23.490821608 +0000 UTC m=+7.757152408" Oct 31 00:27:24.272751 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:24.272291 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nt477" event={"ID":"c63af1f1-ad79-4694-955a-25eb509899d7","Type":"ContainerStarted","Data":"d5bc635991a22b9283a49a822cdc880eda63b43b02826c6c88905012154389dc"} Oct 31 00:27:24.389639 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:24.389090 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nt477" podStartSLOduration=2.3500329349999998 podStartE2EDuration="7.389070008s" totalImagesPullingTime="5.039037073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:24.388335527 +0000 UTC m=+8.654666328" watchObservedRunningTime="2026-10-31 00:27:24.389070008 +0000 UTC m=+8.655400810" Oct 31 00:27:24.389639 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:24.389346 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" podStartSLOduration=7.389336142 podStartE2EDuration="7.389336142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:23.607330272 +0000 UTC m=+7.873661070" watchObservedRunningTime="2026-10-31 00:27:24.389336142 +0000 UTC m=+8.655666948" Oct 31 00:27:25.047432 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:25.047394 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:25.047620 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:25.047470 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:25.047620 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.047573 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:25.047620 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.047608 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:25.047796 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.047651 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs podName:77f7bca8-ea23-4b2b-9eaa-b26425121ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:33.047630865 +0000 UTC m=+17.313961652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs") pod "network-metrics-daemon-mf88n" (UID: "77f7bca8-ea23-4b2b-9eaa-b26425121ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:25.047796 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.047669 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret podName:09934062-de9f-4e88-b25e-4f4a7947e4ce nodeName:}" failed. No retries permitted until 2026-10-31 00:27:33.047662206 +0000 UTC m=+17.313992987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret") pod "global-pull-secret-syncer-ksxrk" (UID: "09934062-de9f-4e88-b25e-4f4a7947e4ce") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:25.147912 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:25.147882 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:25.148091 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.148072 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:25.148161 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.148099 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:25.148161 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.148112 2545 projected.go:196] Error preparing data for projected volume kube-api-access-t8q8z for pod openshift-network-diagnostics/network-check-target-8grlc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:25.148285 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.148178 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z podName:a33cb5d1-e145-47db-9c98-2561a4213883 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:33.148161208 +0000 UTC m=+17.414492000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8q8z" (UniqueName: "kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z") pod "network-check-target-8grlc" (UID: "a33cb5d1-e145-47db-9c98-2561a4213883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:25.210002 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:25.209968 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:25.210156 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.210120 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:25.210222 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:25.210207 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:25.210314 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.210293 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:25.210370 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:25.210357 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:25.210443 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:25.210423 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:26.280159 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:26.280054 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" event={"ID":"fee32755-d814-417f-8311-6a1dc5b60fa5","Type":"ContainerStarted","Data":"b0d09af5a7a25c99d7f89dc63225bee7cf19e92db5e1bbc3c62c52a40ef859ef"} Oct 31 00:27:26.284845 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:26.284086 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fgvdh" event={"ID":"d172f407-b9ac-4193-b0f4-fab1c53e6874","Type":"ContainerStarted","Data":"a511ea22b3aaee28111dcfe00293d65cbc044101aadf95e8e714502a2f5369f7"} Oct 31 00:27:26.400581 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:26.400498 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fgvdh" podStartSLOduration=3.10852705 podStartE2EDuration="9.400481139s" totalImagesPullingTime="6.291954089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:26.400071599 +0000 UTC m=+10.666402412" watchObservedRunningTime="2026-10-31 00:27:26.400481139 +0000 UTC m=+10.666811937" Oct 31 00:27:27.209490 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:27.209072 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:27.209490 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:27.209198 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:27.209490 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:27.209279 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:27.209490 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:27.209355 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:27.209490 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:27.209393 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:27.209490 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:27.209452 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:27.970323 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:27.970298 2545 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:27:28.151923 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.151762 2545 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:27:27.970317726Z","UUID":"ae654f8b-22ea-4903-91a1-a934b867914a","Handler":null,"Name":"","Endpoint":""} Oct 31 00:27:28.154897 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.154873 2545 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:27:28.155089 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.154903 2545 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:27:28.289989 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.289949 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-946fm" event={"ID":"af99842a-26b2-4e57-aae5-3aceef8f0776","Type":"ContainerStarted","Data":"e1108a1c657167345925843c4eab0cdf539430cf4a518028157c3054b3e10443"} Oct 31 00:27:28.292957 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.292929 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" event={"ID":"fee32755-d814-417f-8311-6a1dc5b60fa5","Type":"ContainerStarted","Data":"d095e27e5288403c24147507481588319191916fe0ae01a68adf14db5fd8ce18"} Oct 31 00:27:28.294775 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.294750 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nn2wc" event={"ID":"9316842b-0133-456e-abeb-e0b38e7f04ff","Type":"ContainerStarted","Data":"79cdd6d10b4a35c8fbb7e5455d2e0d84bf15cf6554488c74b778d0013f752426"} Oct 31 00:27:28.509371 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.509321 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nn2wc" podStartSLOduration=2.74792636 podStartE2EDuration="11.509307983s" totalImagesPullingTime="8.761381623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:28.508271853 +0000 UTC m=+12.774602652" watchObservedRunningTime="2026-10-31 00:27:28.509307983 +0000 UTC m=+12.775638778" Oct 31 00:27:28.509640 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.509619 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-946fm" podStartSLOduration=2.439723743 podStartE2EDuration="11.509614134s" totalImagesPullingTime="9.069890391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:28.398523582 +0000 UTC m=+12.664854376" watchObservedRunningTime="2026-10-31 00:27:28.509614134 +0000 UTC m=+12.775944928" Oct 31 00:27:28.589236 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.589210 2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:28.590810 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:28.590392 2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:29.209815 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:29.209770 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:29.210287 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:29.209925 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:29.210359 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:29.210342 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:29.210468 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:29.210444 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:29.210538 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:29.210514 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:29.210592 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:29.210581 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:29.299734 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:29.299501 2545 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="2d77070633117af176681e84584626d81a6757b8786b72804ec175b4836d3863" exitCode=0 Oct 31 00:27:29.299905 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:29.299699 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerDied","Data":"2d77070633117af176681e84584626d81a6757b8786b72804ec175b4836d3863"} Oct 31 00:27:30.304208 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:30.304118 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlblw" event={"ID":"c8057563-4fad-49a2-8d82-4902eca5fa57","Type":"ContainerStarted","Data":"dd9ad9952dfee3e6ee287fbe071799eb9c6a07f87d0b0407030c0b6c28977af5"} Oct 31 00:27:30.305973 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:30.305944 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" event={"ID":"241ccb66-4fd3-427c-9b7f-279439013bfd","Type":"ContainerStarted","Data":"a1667ec659e78325ebeb031d34611bca414d5c5059b2a3caeac55cbefc84416e"} Oct 31 00:27:30.308291 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:30.308263 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" event={"ID":"fee32755-d814-417f-8311-6a1dc5b60fa5","Type":"ContainerStarted","Data":"646ae780c076ecdab5d03435d8dbd8cc036c58c66dd510d56c7cb2508c7b2900"} Oct 31 00:27:30.523822 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:30.523757 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qlblw" podStartSLOduration=4.351615145 podStartE2EDuration="13.523742212s" totalImagesPullingTime="9.172127067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:30.41116477 +0000 UTC m=+14.677495567" watchObservedRunningTime="2026-10-31 00:27:30.523742212 +0000 UTC m=+14.790073006" Oct 31 00:27:30.647643 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:30.647558 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" podStartSLOduration=3.222853148 podStartE2EDuration="13.647542395s" totalImagesPullingTime="10.424689247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:30.523555883 +0000 UTC m=+14.789886678" watchObservedRunningTime="2026-10-31 00:27:30.647542395 +0000 UTC m=+14.913873190" Oct 31 00:27:31.209016 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:31.208942 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:31.209164 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:31.208937 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:31.209164 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:31.209077 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:31.209164 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:31.208949 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:31.209317 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:31.209189 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:31.209317 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:31.209232 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:31.315715 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:31.315671 2545 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="ec9ca3302c2f66387722b85ad8d44379dcb6b0e91be9418fa842404ef8a766f6" exitCode=0 Oct 31 00:27:31.316294 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:31.315747 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerDied","Data":"ec9ca3302c2f66387722b85ad8d44379dcb6b0e91be9418fa842404ef8a766f6"} Oct 31 00:27:31.431284 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:31.431230 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" podStartSLOduration=3.239299198 podStartE2EDuration="14.43121048s" totalImagesPullingTime="11.191911282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:30.649655874 +0000 UTC m=+14.915986670" watchObservedRunningTime="2026-10-31 00:27:31.43121048 +0000 UTC m=+15.697541278" Oct 31 00:27:33.119667 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.119428 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:33.120237 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.119683 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:33.120237 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.119573 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:33.120237 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.119764 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret podName:09934062-de9f-4e88-b25e-4f4a7947e4ce nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.119746768 +0000 UTC m=+33.386077540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret") pod "global-pull-secret-syncer-ksxrk" (UID: "09934062-de9f-4e88-b25e-4f4a7947e4ce") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:33.120237 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.119805 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:33.120237 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.119861 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs podName:77f7bca8-ea23-4b2b-9eaa-b26425121ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.11984638 +0000 UTC m=+33.386177158 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs") pod "network-metrics-daemon-mf88n" (UID: "77f7bca8-ea23-4b2b-9eaa-b26425121ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:33.209276 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.209241 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:33.209432 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.209327 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:33.209487 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.209436 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:33.209487 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.209448 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:33.209577 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.209552 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:33.209670 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.209651 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:33.220463 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.220434 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:33.220600 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.220581 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:33.220600 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.220601 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:33.220693 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.220612 2545 projected.go:196] Error preparing data for projected volume kube-api-access-t8q8z for pod openshift-network-diagnostics/network-check-target-8grlc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:33.220693 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:33.220666 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z podName:a33cb5d1-e145-47db-9c98-2561a4213883 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.220648775 +0000 UTC m=+33.486979572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8q8z" (UniqueName: "kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z") pod "network-check-target-8grlc" (UID: "a33cb5d1-e145-47db-9c98-2561a4213883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:33.323695 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.323647 2545 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="5099a3da421312b4f4f21ed80fabe512064ecb586f76d924fffcb8131692de06" exitCode=0 Oct 31 00:27:33.323866 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.323701 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerDied","Data":"5099a3da421312b4f4f21ed80fabe512064ecb586f76d924fffcb8131692de06"} Oct 31 00:27:33.341429 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.341405 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:33.341523 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.341499 2545 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:27:33.342272 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:33.342246 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nt477" Oct 31 00:27:35.209603 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.209388 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:35.210108 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.209425 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:35.210108 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:35.209702 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:35.210108 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.209458 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:35.210108 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:35.209803 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:35.210108 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:35.209840 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:35.330956 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.330920 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"6476e7269483154e10c1898e25989c482bd33b749dc8e8638591ab45d064bf72"} Oct 31 00:27:35.331146 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.331011 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"c4aaafe890d5330ac81075300a0f067ef0ce1b51faaa2d012689079feaf9ff91"} Oct 31 00:27:35.331146 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.331060 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"74029a7dcab193859d1b1c97cd6e3aec4afcb81f6eae8d44d7da00912f1a472b"} Oct 31 00:27:35.331146 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.331114 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"4e0b278d9fc8a856214683248a29fcc65eda777eb5c3090ae8faa83ce1515180"} Oct 31 00:27:35.331301 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.331160 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"f4e3f862b359f27536e4377cf945787848df16d88edc534521d9cb14257d1da0"} Oct 31 00:27:35.331301 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:35.331209 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"6a0c2c33daedefe34c15074e92133ce0ad4de3b5d2edb857a1b8277eec52da51"} Oct 31 00:27:36.335145 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:36.335054 2545 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="34e61dfdbc07a403031459146690a2a8fbecbc259ca203b97bcc32e19ebb5142" exitCode=0 Oct 31 00:27:36.335145 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:36.335121 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerDied","Data":"34e61dfdbc07a403031459146690a2a8fbecbc259ca203b97bcc32e19ebb5142"} Oct 31 00:27:37.209513 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:37.209481 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:37.209647 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:37.209519 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:37.209647 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:37.209584 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:37.209647 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:37.209600 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:37.209764 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:37.209663 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:37.209764 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:37.209713 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:37.340794 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:37.340707 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"3a8bd2c719cd59ec7cf57d800352ea68f63154213f5a56fded29c533b33f8009"} Oct 31 00:27:37.343691 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:37.343669 2545 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="2a019323adb1e7eb0147412bc95d3f12c26e90b89fa840735f8f3df310ec6d48" exitCode=0 Oct 31 00:27:37.343813 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:37.343704 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerDied","Data":"2a019323adb1e7eb0147412bc95d3f12c26e90b89fa840735f8f3df310ec6d48"} Oct 31 00:27:38.348592 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:38.348556 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" event={"ID":"79c0a7af-0b6e-4056-8e2d-166885cb98dc","Type":"ContainerStarted","Data":"82628ae677fd496abc113892b4fb470e68b7f470f5edbf79a34b120746594296"} Oct 31 00:27:38.556855 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:38.556807 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zz7n4" podStartSLOduration=7.475355716 podStartE2EDuration="21.556772971s" totalImagesPullingTime="14.081417255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:38.510061372 +0000 UTC m=+22.776392167" watchObservedRunningTime="2026-10-31 00:27:38.556772971 +0000 UTC m=+22.823103765" Oct 31 00:27:39.209547 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:39.209506 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:39.209657 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:39.209580 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:39.209657 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:39.209600 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:39.209776 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:39.209697 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:39.209860 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:39.209821 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:39.209945 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:39.209929 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:39.355457 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:39.355214 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" event={"ID":"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa","Type":"ContainerStarted","Data":"bd67c6d94af44eee3aeac4d32fa5b9f598a32818cc7cfa6c9a304fa95d742b2e"} Oct 31 00:27:39.600314 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:39.600263 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" podStartSLOduration=6.962981146 podStartE2EDuration="22.600247849s" totalImagesPullingTime="15.637266703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:39.576404176 +0000 UTC m=+23.842734970" watchObservedRunningTime="2026-10-31 00:27:39.600247849 +0000 UTC m=+23.866578643" Oct 31 00:27:40.357427 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:40.357396 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:40.357984 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:40.357480 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:40.357984 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:40.357523 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:40.372598 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:40.372575 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:40.372725 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:40.372677 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:27:41.209174 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.209146 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:41.209174 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.209173 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:41.209514 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.209188 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:41.209514 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:41.209270 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:41.209514 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:41.209318 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:41.209514 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:41.209387 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:41.898805 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.898756 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ksxrk"] Oct 31 00:27:41.899179 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.898916 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:41.899179 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:41.899002 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:41.901216 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.901188 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8grlc"] Oct 31 00:27:41.901523 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.901294 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:41.901523 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:41.901377 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:41.901971 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.901950 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mf88n"] Oct 31 00:27:41.902063 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:41.902051 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:41.902155 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:41.902135 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:43.208966 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:43.208937 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:43.209369 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:43.208981 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:43.209369 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:43.209097 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:43.209369 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:43.209196 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:44.209581 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:44.209503 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:44.210006 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:44.209623 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:45.209827 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:45.209802 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:45.210146 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:45.209909 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:45.210146 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:45.209959 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:45.210146 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:45.210054 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:46.210057 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:46.210029 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:46.210403 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:46.210115 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mf88n" podUID="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" Oct 31 00:27:47.209105 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.208945 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:47.209223 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.208973 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:47.209223 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.209177 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ksxrk" podUID="09934062-de9f-4e88-b25e-4f4a7947e4ce" Oct 31 00:27:47.209293 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.209265 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8grlc" podUID="a33cb5d1-e145-47db-9c98-2561a4213883" Oct 31 00:27:47.587747 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.587715 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeReady" Oct 31 00:27:47.588129 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.587886 2545 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:27:47.658184 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.658152 2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeReady" Oct 31 00:27:47.696916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.696555 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-24h4x"] Oct 31 00:27:47.698450 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.698431 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n7zhr"] Oct 31 00:27:47.698575 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.698553 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-24h4x" Oct 31 00:27:47.700175 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.700143 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-prfjr"] Oct 31 00:27:47.700271 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.700255 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" Oct 31 00:27:47.701620 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.701602 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c65457c94-5cwm4"] Oct 31 00:27:47.701737 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.701695 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-prfjr" Oct 31 00:27:47.702840 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.702824 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sbz69"] Oct 31 00:27:47.702939 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.702923 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.704045 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.704025 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls"] Oct 31 00:27:47.704199 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.704172 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.705420 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.705404 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.734069 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.734046 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5cfd77cbdf-kcvgr"] Oct 31 00:27:47.735519 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.735505 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4w8gv"] Oct 31 00:27:47.735631 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.735616 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.736964 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.736946 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:47.742764 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.742733 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x"] Oct 31 00:27:47.744452 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.744432 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6"] Oct 31 00:27:47.744547 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.744511 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.746149 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.746132 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl"] Oct 31 00:27:47.746280 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.746264 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" Oct 31 00:27:47.747861 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.747841 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jlcr7"] Oct 31 00:27:47.747961 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.747916 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.748391 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.748372 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.748391 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.748374 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.749903 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.749886 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72"] Oct 31 00:27:47.750036 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.750018 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.751458 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.751442 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b88d6f687-f96vv"] Oct 31 00:27:47.751557 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.751543 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.753376 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.753207 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k"] Oct 31 00:27:47.754097 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.754076 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.754183 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.754174 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:27:47.754264 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.754246 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:27:47.754303 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.754245 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 00:27:47.754922 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.754908 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wqdmn"] Oct 31 00:27:47.755035 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.755020 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" Oct 31 00:27:47.756394 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.756378 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z4v42"] Oct 31 00:27:47.756493 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.756478 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:47.758068 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.758051 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-24h4x"] Oct 31 00:27:47.758153 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.758138 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.779358 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.779338 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rbl9c\"" Oct 31 00:27:47.779489 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.779417 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:27:47.779489 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.779426 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 00:27:47.779489 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.779472 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5xshs\"" Oct 31 00:27:47.779489 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.779488 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-trjc8\"" Oct 31 00:27:47.789236 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.789209 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c65457c94-5cwm4"] Oct 31 00:27:47.790131 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.790113 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n7zhr"] Oct 31 00:27:47.791877 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.791849 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-prfjr"] Oct 31 00:27:47.805766 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.805748 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4w8gv"] Oct 31 00:27:47.814301 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.813537 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sbz69"] Oct 31 00:27:47.819874 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.819854 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72"] Oct 31 00:27:47.820550 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.820530 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x"] Oct 31 00:27:47.821152 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.821135 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl"] Oct 31 00:27:47.821771 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.821755 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cfd77cbdf-kcvgr"] Oct 31 00:27:47.822520 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.822504 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b88d6f687-f96vv"] Oct 31 00:27:47.823125 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.823106 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqdmn"] Oct 31 00:27:47.829926 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.829905 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88lv\" (UniqueName: \"kubernetes.io/projected/a4090b54-5b6a-49f9-b092-d7c309f3d913-kube-api-access-f88lv\") pod \"kas-connection-checker-685895c7c9-n7zhr\" (UID: \"a4090b54-5b6a-49f9-b092-d7c309f3d913\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" Oct 31 00:27:47.830004 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.829932 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgw4\" (UniqueName: \"kubernetes.io/projected/ae509a6d-ba84-42c0-9a51-0e7971cb9d68-kube-api-access-csgw4\") pod \"kas-connection-checker-685895c7c9-24h4x\" (UID: \"ae509a6d-ba84-42c0-9a51-0e7971cb9d68\") " pod="kube-system/kas-connection-checker-685895c7c9-24h4x" Oct 31 00:27:47.830004 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.829951 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcmw\" (UniqueName: \"kubernetes.io/projected/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2-kube-api-access-mjcmw\") pod \"kas-connection-checker-685895c7c9-prfjr\" (UID: \"07e5eb4b-7a3b-43d5-81f9-cda790ace0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-prfjr" Oct 31 00:27:47.830069 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830032 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.830135 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830118 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-snapshots\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.830167 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830149 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14812ec-6d3c-4957-be87-0454c8c21d65-tmp-dir\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.830239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830223 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.830271 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830253 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.830302 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830278 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604f57d5-12dc-4ade-895d-c876688423fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.830333 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830303 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tzb\" (UniqueName: \"kubernetes.io/projected/b634720f-79f7-4d34-a971-8d1ba231b3d3-kube-api-access-m5tzb\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.830333 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830326 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-stats-auth\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.830427 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830353 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwt9\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.830427 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830413 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.830524 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830443 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.830524 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830460 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:47.830524 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830478 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.830524 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830492 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-cert\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:47.830640 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830533 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sz7n\" (UniqueName: \"kubernetes.io/projected/53a51104-b6b6-4c74-a3c0-e2c5c378f28f-kube-api-access-9sz7n\") pod \"volume-data-source-validator-58c9f5f46d-86qq6\" (UID: \"53a51104-b6b6-4c74-a3c0-e2c5c378f28f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" Oct 31 00:27:47.830640 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830602 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rwh6\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.830640 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830622 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:47.830728 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830661 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.830728 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830681 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.830728 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830710 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38807dac-4dc9-4d27-932b-e81189808eb1-serving-cert\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.830831 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830770 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38807dac-4dc9-4d27-932b-e81189808eb1-config\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.830831 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830803 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-config\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.830831 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830824 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.830920 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830840 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqq5\" (UniqueName: \"kubernetes.io/projected/7c0f42a0-fd35-4121-96b9-1d6363689977-kube-api-access-lhqq5\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.830920 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830855 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fz89\" (UniqueName: \"kubernetes.io/projected/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-kube-api-access-4fz89\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:47.830920 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830874 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.830920 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830904 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.831033 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830970 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.831033 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.830996 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c0f42a0-fd35-4121-96b9-1d6363689977-serving-cert\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.831102 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831028 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.831102 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831063 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jxj\" (UniqueName: \"kubernetes.io/projected/f4428250-46e8-4f67-bb0c-285938ee2175-kube-api-access-f8jxj\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.831102 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831094 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.831230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831117 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5tkt\" (UniqueName: \"kubernetes.io/projected/f14812ec-6d3c-4957-be87-0454c8c21d65-kube-api-access-g5tkt\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.831230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831160 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpcz\" (UniqueName: \"kubernetes.io/projected/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1-kube-api-access-cwpcz\") pod \"network-check-source-6f78d59f99-cz22k\" (UID: \"c1caab5f-35c5-4f4d-8f81-0d18527a7ef1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" Oct 31 00:27:47.831230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831177 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjb6d\" (UniqueName: \"kubernetes.io/projected/341fbe38-f97c-4a00-938b-6c4b9a30197e-kube-api-access-kjb6d\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.831230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831198 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-default-certificate\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.831230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831215 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.831365 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831237 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.831365 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831272 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/604f57d5-12dc-4ade-895d-c876688423fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.831365 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831316 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-tmp\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.831365 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831340 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/341fbe38-f97c-4a00-938b-6c4b9a30197e-serving-cert\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.831365 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831355 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f14812ec-6d3c-4957-be87-0454c8c21d65-config-volume\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.831504 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831400 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.831504 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831422 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjvm\" (UniqueName: \"kubernetes.io/projected/38807dac-4dc9-4d27-932b-e81189808eb1-kube-api-access-lkjvm\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.831504 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831437 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.831504 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831470 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpfb\" (UniqueName: \"kubernetes.io/projected/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-kube-api-access-8rpfb\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.831612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831509 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4428250-46e8-4f67-bb0c-285938ee2175-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.831612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831528 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.831612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831549 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.831612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831573 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.831612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831589 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.831835 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831613 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-service-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.831835 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.831634 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pfnd\" (UniqueName: \"kubernetes.io/projected/604f57d5-12dc-4ade-895d-c876688423fb-kube-api-access-5pfnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.854062 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.853998 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls"] Oct 31 00:27:47.854434 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.854413 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 00:27:47.857772 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.857724 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.857875 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.857853 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:27:47.857969 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.857949 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.858034 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.857973 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 00:27:47.858034 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.857980 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.858132 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858078 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.858394 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858366 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.858394 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858366 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.858394 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858368 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 00:27:47.858562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858405 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.858562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858464 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.858647 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858595 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 00:27:47.858647 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.858597 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 00:27:47.860007 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.859989 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.898719 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.898692 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.900123 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900102 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 00:27:47.900230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900174 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.900230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900217 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 00:27:47.900322 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900232 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.900322 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900248 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.900684 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900659 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.900835 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900766 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.900901 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900841 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nl57f\"" Oct 31 00:27:47.900901 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900819 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 00:27:47.900901 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900854 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.900901 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900810 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:27:47.901091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900931 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 00:27:47.901091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900961 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 00:27:47.901091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900980 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:47.901091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900987 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 00:27:47.901091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901039 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:27:47.901091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.900934 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mpt2z\"" Oct 31 00:27:47.901091 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901066 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901208 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-c9dtf\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901231 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901234 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901250 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901272 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jw542\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901285 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901287 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sqldj\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901297 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901309 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901316 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 00:27:47.901386 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901316 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r5jc7\"" Oct 31 00:27:47.901833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901466 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-65vz6\"" Oct 31 00:27:47.901833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901506 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:27:47.901833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901589 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:27:47.901833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901668 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hg2s2\"" Oct 31 00:27:47.901833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901728 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52f7q\"" Oct 31 00:27:47.901833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901817 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ghdng\"" Oct 31 00:27:47.902017 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.901911 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jlcr7"] Oct 31 00:27:47.904315 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.904299 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:27:47.905122 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.905106 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4v42"] Oct 31 00:27:47.905752 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.905736 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6"] Oct 31 00:27:47.906402 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.906384 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k"] Oct 31 00:27:47.932286 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932263 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-service-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.932378 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932298 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pfnd\" (UniqueName: \"kubernetes.io/projected/604f57d5-12dc-4ade-895d-c876688423fb-kube-api-access-5pfnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.932378 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932325 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88lv\" (UniqueName: \"kubernetes.io/projected/a4090b54-5b6a-49f9-b092-d7c309f3d913-kube-api-access-f88lv\") pod \"kas-connection-checker-685895c7c9-n7zhr\" (UID: \"a4090b54-5b6a-49f9-b092-d7c309f3d913\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" Oct 31 00:27:47.932378 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932357 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgw4\" (UniqueName: \"kubernetes.io/projected/ae509a6d-ba84-42c0-9a51-0e7971cb9d68-kube-api-access-csgw4\") pod \"kas-connection-checker-685895c7c9-24h4x\" (UID: \"ae509a6d-ba84-42c0-9a51-0e7971cb9d68\") " pod="kube-system/kas-connection-checker-685895c7c9-24h4x" Oct 31 00:27:47.932519 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932382 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcmw\" (UniqueName: \"kubernetes.io/projected/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2-kube-api-access-mjcmw\") pod \"kas-connection-checker-685895c7c9-prfjr\" (UID: \"07e5eb4b-7a3b-43d5-81f9-cda790ace0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-prfjr" Oct 31 00:27:47.932519 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932410 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.932519 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932457 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-snapshots\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.932519 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932480 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14812ec-6d3c-4957-be87-0454c8c21d65-tmp-dir\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.932711 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932523 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.932711 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932550 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.932711 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932586 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604f57d5-12dc-4ade-895d-c876688423fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.932711 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.932607 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:47.932711 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.932624 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b88d6f687-f96vv: secret "image-registry-tls" not found Oct 31 00:27:47.932711 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932611 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tzb\" (UniqueName: \"kubernetes.io/projected/b634720f-79f7-4d34-a971-8d1ba231b3d3-kube-api-access-m5tzb\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.932711 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.932691 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls podName:c998bbec-e9a8-45b9-8475-50a4e4fb2c35 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.432671476 +0000 UTC m=+32.699002267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls") pod "image-registry-5b88d6f687-f96vv" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35") : secret "image-registry-tls" not found Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932735 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-stats-auth\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932817 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14812ec-6d3c-4957-be87-0454c8c21d65-tmp-dir\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932853 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clwt9\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932887 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932923 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932948 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.932984 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933007 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-cert\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933033 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sz7n\" (UniqueName: \"kubernetes.io/projected/53a51104-b6b6-4c74-a3c0-e2c5c378f28f-kube-api-access-9sz7n\") pod \"volume-data-source-validator-58c9f5f46d-86qq6\" (UID: \"53a51104-b6b6-4c74-a3c0-e2c5c378f28f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" Oct 31 00:27:47.933065 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933067 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwh6\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933096 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933123 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-service-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933126 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933153 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933163 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933182 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38807dac-4dc9-4d27-932b-e81189808eb1-serving-cert\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933225 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38807dac-4dc9-4d27-932b-e81189808eb1-config\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933250 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-config\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933163 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-snapshots\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933289 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933314 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqq5\" (UniqueName: \"kubernetes.io/projected/7c0f42a0-fd35-4121-96b9-1d6363689977-kube-api-access-lhqq5\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933332 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604f57d5-12dc-4ade-895d-c876688423fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.933254 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:27:47.933529 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.933535 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert podName:48eb4acb-8de7-4b4e-9fea-ca7bf8556014 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.433518826 +0000 UTC m=+32.699849620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4w8gv" (UID: "48eb4acb-8de7-4b4e-9fea-ca7bf8556014") : secret "networking-console-plugin-cert" not found Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.933322 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.933579 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls podName:b634720f-79f7-4d34-a971-8d1ba231b3d3 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.433569942 +0000 UTC m=+32.699900746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5h7kl" (UID: "b634720f-79f7-4d34-a971-8d1ba231b3d3") : secret "samples-operator-tls" not found Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.933997 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934003 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934045 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fz89\" (UniqueName: \"kubernetes.io/projected/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-kube-api-access-4fz89\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934073 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934098 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934123 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-config\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934135 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934156 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c0f42a0-fd35-4121-96b9-1d6363689977-serving-cert\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934177 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934200 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jxj\" (UniqueName: \"kubernetes.io/projected/f4428250-46e8-4f67-bb0c-285938ee2175-kube-api-access-f8jxj\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.934208 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:27:47.934221 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934224 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934248 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tkt\" (UniqueName: \"kubernetes.io/projected/f14812ec-6d3c-4957-be87-0454c8c21d65-kube-api-access-g5tkt\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.934255 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls podName:f14812ec-6d3c-4957-be87-0454c8c21d65 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.434241639 +0000 UTC m=+32.700572431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls") pod "dns-default-z4v42" (UID: "f14812ec-6d3c-4957-be87-0454c8c21d65") : secret "dns-default-metrics-tls" not found Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934254 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934293 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpcz\" (UniqueName: \"kubernetes.io/projected/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1-kube-api-access-cwpcz\") pod \"network-check-source-6f78d59f99-cz22k\" (UID: \"c1caab5f-35c5-4f4d-8f81-0d18527a7ef1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934318 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjb6d\" (UniqueName: \"kubernetes.io/projected/341fbe38-f97c-4a00-938b-6c4b9a30197e-kube-api-access-kjb6d\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934341 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-default-certificate\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934364 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934385 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934412 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/604f57d5-12dc-4ade-895d-c876688423fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934437 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-tmp\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934453 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38807dac-4dc9-4d27-932b-e81189808eb1-config\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.934595 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.434581338 +0000 UTC m=+32.700912134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : configmap references non-existent config key: service-ca.crt Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934596 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.934904 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.934674 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-tmp\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935597 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/341fbe38-f97c-4a00-938b-6c4b9a30197e-serving-cert\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935633 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f14812ec-6d3c-4957-be87-0454c8c21d65-config-volume\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.935649 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.935662 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c65457c94-5cwm4: secret "image-registry-tls" not found Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935665 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935692 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjvm\" (UniqueName: \"kubernetes.io/projected/38807dac-4dc9-4d27-932b-e81189808eb1-kube-api-access-lkjvm\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.935700 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:27:47.935804 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.935704 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls podName:8fcf1ba8-6176-418b-8477-6f5b89a16a41 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.435689807 +0000 UTC m=+32.702020588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls") pod "image-registry-7c65457c94-5cwm4" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41") : secret "image-registry-tls" not found Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.935814 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.435778562 +0000 UTC m=+32.702109335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : secret "router-metrics-certs-default" not found Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935833 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935866 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpfb\" (UniqueName: \"kubernetes.io/projected/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-kube-api-access-8rpfb\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935898 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4428250-46e8-4f67-bb0c-285938ee2175-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935925 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935931 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935953 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.935989 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.936018 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.936239 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.936225 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f14812ec-6d3c-4957-be87-0454c8c21d65-config-volume\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.937111 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.936500 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.938659 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.937367 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.938659 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.938107 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.938659 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.938394 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.938868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.938674 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.938868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.938687 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-stats-auth\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.938868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.938743 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.939019 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.938999 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.939087 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.939071 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:47.939144 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:47.939111 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls podName:f4428250-46e8-4f67-bb0c-285938ee2175 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:48.439101254 +0000 UTC m=+32.705432027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlk72" (UID: "f4428250-46e8-4f67-bb0c-285938ee2175") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:47.939206 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.939147 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-cert\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:47.939421 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.939392 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38807dac-4dc9-4d27-932b-e81189808eb1-serving-cert\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:47.939486 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.939440 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4428250-46e8-4f67-bb0c-285938ee2175-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:47.939918 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.939896 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c0f42a0-fd35-4121-96b9-1d6363689977-serving-cert\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.940355 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.940331 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-default-certificate\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:47.940994 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.940975 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/341fbe38-f97c-4a00-938b-6c4b9a30197e-serving-cert\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:47.941246 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.941231 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/604f57d5-12dc-4ade-895d-c876688423fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.941632 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.941616 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.994945 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.994913 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pfnd\" (UniqueName: \"kubernetes.io/projected/604f57d5-12dc-4ade-895d-c876688423fb-kube-api-access-5pfnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:47.995396 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.995335 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tkt\" (UniqueName: \"kubernetes.io/projected/f14812ec-6d3c-4957-be87-0454c8c21d65-kube-api-access-g5tkt\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:47.995396 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.995374 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.995396 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.995385 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88lv\" (UniqueName: \"kubernetes.io/projected/a4090b54-5b6a-49f9-b092-d7c309f3d913-kube-api-access-f88lv\") pod \"kas-connection-checker-685895c7c9-n7zhr\" (UID: \"a4090b54-5b6a-49f9-b092-d7c309f3d913\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" Oct 31 00:27:47.995775 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.995750 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgw4\" (UniqueName: \"kubernetes.io/projected/ae509a6d-ba84-42c0-9a51-0e7971cb9d68-kube-api-access-csgw4\") pod \"kas-connection-checker-685895c7c9-24h4x\" (UID: \"ae509a6d-ba84-42c0-9a51-0e7971cb9d68\") " pod="kube-system/kas-connection-checker-685895c7c9-24h4x" Oct 31 00:27:47.995882 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.995774 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwh6\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:47.996289 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.996270 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqq5\" (UniqueName: \"kubernetes.io/projected/7c0f42a0-fd35-4121-96b9-1d6363689977-kube-api-access-lhqq5\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:47.996705 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.996683 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcmw\" (UniqueName: \"kubernetes.io/projected/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2-kube-api-access-mjcmw\") pod \"kas-connection-checker-685895c7c9-prfjr\" (UID: \"07e5eb4b-7a3b-43d5-81f9-cda790ace0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-prfjr" Oct 31 00:27:47.997027 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.997007 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:47.997336 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.997318 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sz7n\" (UniqueName: \"kubernetes.io/projected/53a51104-b6b6-4c74-a3c0-e2c5c378f28f-kube-api-access-9sz7n\") pod \"volume-data-source-validator-58c9f5f46d-86qq6\" (UID: \"53a51104-b6b6-4c74-a3c0-e2c5c378f28f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" Oct 31 00:27:47.997626 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:47.997608 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tzb\" (UniqueName: \"kubernetes.io/projected/b634720f-79f7-4d34-a971-8d1ba231b3d3-kube-api-access-m5tzb\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:48.000038 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.000012 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwt9\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:48.001413 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.001388 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jxj\" (UniqueName: \"kubernetes.io/projected/f4428250-46e8-4f67-bb0c-285938ee2175-kube-api-access-f8jxj\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:48.001413 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.001407 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fz89\" (UniqueName: \"kubernetes.io/projected/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-kube-api-access-4fz89\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:48.001556 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.001528 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjb6d\" (UniqueName: \"kubernetes.io/projected/341fbe38-f97c-4a00-938b-6c4b9a30197e-kube-api-access-kjb6d\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:48.001746 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.001728 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpcz\" (UniqueName: \"kubernetes.io/projected/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1-kube-api-access-cwpcz\") pod \"network-check-source-6f78d59f99-cz22k\" (UID: \"c1caab5f-35c5-4f4d-8f81-0d18527a7ef1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" Oct 31 00:27:48.001835 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.001732 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjvm\" (UniqueName: \"kubernetes.io/projected/38807dac-4dc9-4d27-932b-e81189808eb1-kube-api-access-lkjvm\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:48.002058 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.002042 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpfb\" (UniqueName: \"kubernetes.io/projected/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-kube-api-access-8rpfb\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:48.009927 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.009907 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-24h4x" Oct 31 00:27:48.016928 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.016910 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" Oct 31 00:27:48.024937 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.024918 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-prfjr" Oct 31 00:27:48.036775 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.036752 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:27:48.043187 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.043144 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:27:48.063644 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.063416 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:27:48.073106 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.072905 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" Oct 31 00:27:48.088325 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.088222 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:48.112017 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.111989 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" Oct 31 00:27:48.116155 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.116132 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:27:48.225535 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.211276 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:48.260562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.260536 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:27:48.260562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.260545 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m2wdr\"" Oct 31 00:27:48.394820 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.394727 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n7zhr"] Oct 31 00:27:48.397942 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.397883 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-prfjr"] Oct 31 00:27:48.398256 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.398235 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4090b54_5b6a_49f9_b092_d7c309f3d913.slice/crio-a1e48c120d9af5509cb12c28d135e33e479156741fcbd392956ae5bab771233f WatchSource:0}: Error finding container a1e48c120d9af5509cb12c28d135e33e479156741fcbd392956ae5bab771233f: Status 404 returned error can't find the container with id a1e48c120d9af5509cb12c28d135e33e479156741fcbd392956ae5bab771233f Oct 31 00:27:48.400588 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.400569 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e5eb4b_7a3b_43d5_81f9_cda790ace0b2.slice/crio-f67baedb4acce39a4212f5fdc7726fbcd15f0ab2cc8b6b5ff968137c8486a995 WatchSource:0}: Error finding container f67baedb4acce39a4212f5fdc7726fbcd15f0ab2cc8b6b5ff968137c8486a995: Status 404 returned error can't find the container with id f67baedb4acce39a4212f5fdc7726fbcd15f0ab2cc8b6b5ff968137c8486a995 Oct 31 00:27:48.401182 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.401141 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-24h4x"] Oct 31 00:27:48.404172 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.404152 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae509a6d_ba84_42c0_9a51_0e7971cb9d68.slice/crio-ceda5fc6e9c6871ca5ca5788ad24b6b065aef2b7ac53bdc9f4763314b2cecf15 WatchSource:0}: Error finding container ceda5fc6e9c6871ca5ca5788ad24b6b065aef2b7ac53bdc9f4763314b2cecf15: Status 404 returned error can't find the container with id ceda5fc6e9c6871ca5ca5788ad24b6b065aef2b7ac53bdc9f4763314b2cecf15 Oct 31 00:27:48.442916 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.442872 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sbz69"] Oct 31 00:27:48.446069 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.446046 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls"] Oct 31 00:27:48.452825 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.452795 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0f42a0_fd35_4121_96b9_1d6363689977.slice/crio-98bdb3de1e11569289361a6f2d17a2dd9c2753abafbf9349abbd09c33ff010a7 WatchSource:0}: Error finding container 98bdb3de1e11569289361a6f2d17a2dd9c2753abafbf9349abbd09c33ff010a7: Status 404 returned error can't find the container with id 98bdb3de1e11569289361a6f2d17a2dd9c2753abafbf9349abbd09c33ff010a7 Oct 31 00:27:48.454854 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.454797 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604f57d5_12dc_4ade_895d_c876688423fb.slice/crio-21e4e1e7dab09e551ef7ef8b3a6d1c8b0a0de1a66aa69ce85ea84b2ec9237b8c WatchSource:0}: Error finding container 21e4e1e7dab09e551ef7ef8b3a6d1c8b0a0de1a66aa69ce85ea84b2ec9237b8c: Status 404 returned error can't find the container with id 21e4e1e7dab09e551ef7ef8b3a6d1c8b0a0de1a66aa69ce85ea84b2ec9237b8c Oct 31 00:27:48.454944 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.454885 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:48.454944 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.454934 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:48.455118 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.454968 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:48.455118 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.454994 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:48.455118 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.455039 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:48.455118 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.455082 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:48.455390 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.455136 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:48.455390 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.455187 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:48.455390 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455316 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:27:48.455390 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455367 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert podName:48eb4acb-8de7-4b4e-9fea-ca7bf8556014 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.455350786 +0000 UTC m=+33.721681561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4w8gv" (UID: "48eb4acb-8de7-4b4e-9fea-ca7bf8556014") : secret "networking-console-plugin-cert" not found Oct 31 00:27:48.455590 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455436 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:27:48.455590 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455468 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls podName:f14812ec-6d3c-4957-be87-0454c8c21d65 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.455457892 +0000 UTC m=+33.721788666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls") pod "dns-default-z4v42" (UID: "f14812ec-6d3c-4957-be87-0454c8c21d65") : secret "dns-default-metrics-tls" not found Oct 31 00:27:48.455590 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455533 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.455522714 +0000 UTC m=+33.721853487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : configmap references non-existent config key: service-ca.crt Oct 31 00:27:48.455590 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455586 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:27:48.455840 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455616 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.455606174 +0000 UTC m=+33.721936961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : secret "router-metrics-certs-default" not found Oct 31 00:27:48.455840 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455673 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:48.455840 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455685 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c65457c94-5cwm4: secret "image-registry-tls" not found Oct 31 00:27:48.455840 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455714 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls podName:8fcf1ba8-6176-418b-8477-6f5b89a16a41 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.45570575 +0000 UTC m=+33.722036526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls") pod "image-registry-7c65457c94-5cwm4" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41") : secret "image-registry-tls" not found Oct 31 00:27:48.455840 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455769 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:48.455840 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455820 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls podName:f4428250-46e8-4f67-bb0c-285938ee2175 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.455807636 +0000 UTC m=+33.722138409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlk72" (UID: "f4428250-46e8-4f67-bb0c-285938ee2175") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:48.456131 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455879 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:27:48.456131 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455912 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls podName:b634720f-79f7-4d34-a971-8d1ba231b3d3 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.455902255 +0000 UTC m=+33.722233028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5h7kl" (UID: "b634720f-79f7-4d34-a971-8d1ba231b3d3") : secret "samples-operator-tls" not found Oct 31 00:27:48.456131 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455964 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:48.456131 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.455975 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b88d6f687-f96vv: secret "image-registry-tls" not found Oct 31 00:27:48.456131 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:48.456002 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls podName:c998bbec-e9a8-45b9-8475-50a4e4fb2c35 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:49.455993694 +0000 UTC m=+33.722324468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls") pod "image-registry-5b88d6f687-f96vv" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35") : secret "image-registry-tls" not found Oct 31 00:27:48.545305 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.545222 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6"] Oct 31 00:27:48.551331 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.551303 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k"] Oct 31 00:27:48.551406 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.551369 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jlcr7"] Oct 31 00:27:48.553993 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.553633 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x"] Oct 31 00:27:48.556911 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:48.554663 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqdmn"] Oct 31 00:27:48.556911 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.554691 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a51104_b6b6_4c74_a3c0_e2c5c378f28f.slice/crio-562095794d42c600a66f4491df56ef216cf5866a1b6c05240d0ea566189371d7 WatchSource:0}: Error finding container 562095794d42c600a66f4491df56ef216cf5866a1b6c05240d0ea566189371d7: Status 404 returned error can't find the container with id 562095794d42c600a66f4491df56ef216cf5866a1b6c05240d0ea566189371d7 Oct 31 00:27:48.556911 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.555261 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a345f30_ddf3_478a_81e0_5d50cbf38dc4.slice/crio-e8b4d5b4eaa16a4737e886726db99da7327961d4b9d5dd0533cdc53f4e5f3f70 WatchSource:0}: Error finding container e8b4d5b4eaa16a4737e886726db99da7327961d4b9d5dd0533cdc53f4e5f3f70: Status 404 returned error can't find the container with id e8b4d5b4eaa16a4737e886726db99da7327961d4b9d5dd0533cdc53f4e5f3f70 Oct 31 00:27:48.556911 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.555688 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38807dac_4dc9_4d27_932b_e81189808eb1.slice/crio-0a4ff567ada319cd933ca254ed30e29c278c7c6bbb3367587b5905b3f8e22b23 WatchSource:0}: Error finding container 0a4ff567ada319cd933ca254ed30e29c278c7c6bbb3367587b5905b3f8e22b23: Status 404 returned error can't find the container with id 0a4ff567ada319cd933ca254ed30e29c278c7c6bbb3367587b5905b3f8e22b23 Oct 31 00:27:48.556911 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.556675 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341fbe38_f97c_4a00_938b_6c4b9a30197e.slice/crio-238ed5358e983b13e1f4e6c70c23cd89186d0980ed12c716297a0c2227b78555 WatchSource:0}: Error finding container 238ed5358e983b13e1f4e6c70c23cd89186d0980ed12c716297a0c2227b78555: Status 404 returned error can't find the container with id 238ed5358e983b13e1f4e6c70c23cd89186d0980ed12c716297a0c2227b78555 Oct 31 00:27:48.557367 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:48.557234 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1caab5f_35c5_4f4d_8f81_0d18527a7ef1.slice/crio-bb593b0969779537fd541df460e37740c48c8dbbf3f77bcb14c77cac2cf76046 WatchSource:0}: Error finding container bb593b0969779537fd541df460e37740c48c8dbbf3f77bcb14c77cac2cf76046: Status 404 returned error can't find the container with id bb593b0969779537fd541df460e37740c48c8dbbf3f77bcb14c77cac2cf76046 Oct 31 00:27:49.161990 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.161958 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:27:49.162366 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.162089 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:49.162366 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.162137 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:27:49.162366 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.162209 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs podName:77f7bca8-ea23-4b2b-9eaa-b26425121ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:21.16218468 +0000 UTC m=+65.428515471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs") pod "network-metrics-daemon-mf88n" (UID: "77f7bca8-ea23-4b2b-9eaa-b26425121ff5") : secret "metrics-daemon-secret" not found Oct 31 00:27:49.162366 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.162217 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:49.162366 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.162270 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret podName:09934062-de9f-4e88-b25e-4f4a7947e4ce nodeName:}" failed. No retries permitted until 2026-10-31 00:28:21.162255817 +0000 UTC m=+65.428586591 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret") pod "global-pull-secret-syncer-ksxrk" (UID: "09934062-de9f-4e88-b25e-4f4a7947e4ce") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:49.208980 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.208944 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:49.208980 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.208978 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:27:49.257868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.257842 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:27:49.257868 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.257845 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nwzth\"" Oct 31 00:27:49.257984 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.257865 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dwmlr\"" Oct 31 00:27:49.262709 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.262682 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:49.265442 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.265418 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:49.377340 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.377301 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" event={"ID":"c1caab5f-35c5-4f4d-8f81-0d18527a7ef1","Type":"ContainerStarted","Data":"bb593b0969779537fd541df460e37740c48c8dbbf3f77bcb14c77cac2cf76046"} Oct 31 00:27:49.378374 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.378352 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" event={"ID":"341fbe38-f97c-4a00-938b-6c4b9a30197e","Type":"ContainerStarted","Data":"238ed5358e983b13e1f4e6c70c23cd89186d0980ed12c716297a0c2227b78555"} Oct 31 00:27:49.381866 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.381846 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" event={"ID":"a4090b54-5b6a-49f9-b092-d7c309f3d913","Type":"ContainerStarted","Data":"d35b289675cce2b2bc81eae82f5ba4f392a63589148baccf3367d40b282a9029"} Oct 31 00:27:49.382013 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.381876 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" event={"ID":"a4090b54-5b6a-49f9-b092-d7c309f3d913","Type":"ContainerStarted","Data":"a1e48c120d9af5509cb12c28d135e33e479156741fcbd392956ae5bab771233f"} Oct 31 00:27:49.383016 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.382998 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqdmn" event={"ID":"7a345f30-ddf3-478a-81e0-5d50cbf38dc4","Type":"ContainerStarted","Data":"e8b4d5b4eaa16a4737e886726db99da7327961d4b9d5dd0533cdc53f4e5f3f70"} Oct 31 00:27:49.384036 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.384015 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" event={"ID":"53a51104-b6b6-4c74-a3c0-e2c5c378f28f","Type":"ContainerStarted","Data":"562095794d42c600a66f4491df56ef216cf5866a1b6c05240d0ea566189371d7"} Oct 31 00:27:49.384993 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.384976 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" event={"ID":"604f57d5-12dc-4ade-895d-c876688423fb","Type":"ContainerStarted","Data":"21e4e1e7dab09e551ef7ef8b3a6d1c8b0a0de1a66aa69ce85ea84b2ec9237b8c"} Oct 31 00:27:49.386285 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.386257 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-24h4x" event={"ID":"ae509a6d-ba84-42c0-9a51-0e7971cb9d68","Type":"ContainerStarted","Data":"a9cd7b516219f3434019970e681086b2d03f1d8fbd704abfe1088e8b975cd9e4"} Oct 31 00:27:49.386373 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.386296 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-24h4x" event={"ID":"ae509a6d-ba84-42c0-9a51-0e7971cb9d68","Type":"ContainerStarted","Data":"ceda5fc6e9c6871ca5ca5788ad24b6b065aef2b7ac53bdc9f4763314b2cecf15"} Oct 31 00:27:49.387413 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.387391 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" event={"ID":"38807dac-4dc9-4d27-932b-e81189808eb1","Type":"ContainerStarted","Data":"0a4ff567ada319cd933ca254ed30e29c278c7c6bbb3367587b5905b3f8e22b23"} Oct 31 00:27:49.388473 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.388452 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sbz69" event={"ID":"7c0f42a0-fd35-4121-96b9-1d6363689977","Type":"ContainerStarted","Data":"98bdb3de1e11569289361a6f2d17a2dd9c2753abafbf9349abbd09c33ff010a7"} Oct 31 00:27:49.389687 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.389670 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-prfjr" event={"ID":"07e5eb4b-7a3b-43d5-81f9-cda790ace0b2","Type":"ContainerStarted","Data":"efcef1894395f40c3d2f4e0582183e3e11ebe6acc269d618cd12b69e07337e1a"} Oct 31 00:27:49.389748 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.389700 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-prfjr" event={"ID":"07e5eb4b-7a3b-43d5-81f9-cda790ace0b2","Type":"ContainerStarted","Data":"f67baedb4acce39a4212f5fdc7726fbcd15f0ab2cc8b6b5ff968137c8486a995"} Oct 31 00:27:49.464617 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.464591 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:49.464733 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.464633 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:49.464800 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.464739 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:49.464800 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.464738 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:27:49.464901 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.464813 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:49.464901 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.464828 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls podName:b634720f-79f7-4d34-a971-8d1ba231b3d3 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.464814549 +0000 UTC m=+35.731145322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5h7kl" (UID: "b634720f-79f7-4d34-a971-8d1ba231b3d3") : secret "samples-operator-tls" not found Oct 31 00:27:49.464901 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.464830 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b88d6f687-f96vv: secret "image-registry-tls" not found Oct 31 00:27:49.464901 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.464851 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:27:49.464901 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.464873 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls podName:c998bbec-e9a8-45b9-8475-50a4e4fb2c35 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.464859958 +0000 UTC m=+35.731190759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls") pod "image-registry-5b88d6f687-f96vv" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35") : secret "image-registry-tls" not found Oct 31 00:27:49.465128 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.464907 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert podName:48eb4acb-8de7-4b4e-9fea-ca7bf8556014 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.464892237 +0000 UTC m=+35.731223016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4w8gv" (UID: "48eb4acb-8de7-4b4e-9fea-ca7bf8556014") : secret "networking-console-plugin-cert" not found Oct 31 00:27:49.465128 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.464932 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:49.465128 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.464982 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:49.465128 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465064 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:27:49.465128 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465083 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.465071014 +0000 UTC m=+35.731401801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : configmap references non-existent config key: service-ca.crt Oct 31 00:27:49.465128 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465103 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls podName:f14812ec-6d3c-4957-be87-0454c8c21d65 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.465092687 +0000 UTC m=+35.731423464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls") pod "dns-default-z4v42" (UID: "f14812ec-6d3c-4957-be87-0454c8c21d65") : secret "dns-default-metrics-tls" not found Oct 31 00:27:49.465128 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.465126 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.465147 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.465176 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465235 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465244 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465254 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c65457c94-5cwm4: secret "image-registry-tls" not found Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465260 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.465252904 +0000 UTC m=+35.731583692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : secret "router-metrics-certs-default" not found Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465245 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465281 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls podName:8fcf1ba8-6176-418b-8477-6f5b89a16a41 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.46527288 +0000 UTC m=+35.731603656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls") pod "image-registry-7c65457c94-5cwm4" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41") : secret "image-registry-tls" not found Oct 31 00:27:49.465367 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:49.465308 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls podName:f4428250-46e8-4f67-bb0c-285938ee2175 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:51.465297202 +0000 UTC m=+35.731627982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlk72" (UID: "f4428250-46e8-4f67-bb0c-285938ee2175") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:49.521622 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.521594 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:49.704284 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.704231 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" podStartSLOduration=403.704216776 podStartE2EDuration="6m43.704216776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:49.700867586 +0000 UTC m=+33.967198382" watchObservedRunningTime="2026-10-31 00:27:49.704216776 +0000 UTC m=+33.970547564" Oct 31 00:27:49.809399 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.807741 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-24h4x" podStartSLOduration=403.807724697 podStartE2EDuration="6m43.807724697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:49.807272269 +0000 UTC m=+34.073603066" watchObservedRunningTime="2026-10-31 00:27:49.807724697 +0000 UTC m=+34.074055498" Oct 31 00:27:49.905911 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.905558 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8grlc"] Oct 31 00:27:49.930963 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:49.930928 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33cb5d1_e145_47db_9c98_2561a4213883.slice/crio-0d77d2c8b3f59aca5189494c2aae27361a48344f3aca5896afe11a935a00b979 WatchSource:0}: Error finding container 0d77d2c8b3f59aca5189494c2aae27361a48344f3aca5896afe11a935a00b979: Status 404 returned error can't find the container with id 0d77d2c8b3f59aca5189494c2aae27361a48344f3aca5896afe11a935a00b979 Oct 31 00:27:49.931884 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:49.931826 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-prfjr" podStartSLOduration=403.931807392 podStartE2EDuration="6m43.931807392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:49.930774189 +0000 UTC m=+34.197104986" watchObservedRunningTime="2026-10-31 00:27:49.931807392 +0000 UTC m=+34.198138186" Oct 31 00:27:50.396825 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:50.396795 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8grlc" event={"ID":"a33cb5d1-e145-47db-9c98-2561a4213883","Type":"ContainerStarted","Data":"0d77d2c8b3f59aca5189494c2aae27361a48344f3aca5896afe11a935a00b979"} Oct 31 00:27:51.407051 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.406971 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" event={"ID":"53a51104-b6b6-4c74-a3c0-e2c5c378f28f","Type":"ContainerStarted","Data":"5faa624e478fd77232891544b41de628e7e2ccf73fa1996d5f6e731eaec383fa"} Oct 31 00:27:51.487455 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487416 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:51.487618 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487485 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:51.487618 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487525 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:51.487618 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487555 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:51.487618 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487599 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:51.487844 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487632 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:51.487844 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487665 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:51.487844 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.487696 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:51.487844 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.487832 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:27:51.488038 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.487883 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert podName:48eb4acb-8de7-4b4e-9fea-ca7bf8556014 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.487868601 +0000 UTC m=+39.754199374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4w8gv" (UID: "48eb4acb-8de7-4b4e-9fea-ca7bf8556014") : secret "networking-console-plugin-cert" not found Oct 31 00:27:51.488292 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488271 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:27:51.488374 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488329 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls podName:f14812ec-6d3c-4957-be87-0454c8c21d65 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.488312963 +0000 UTC m=+39.754643750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls") pod "dns-default-z4v42" (UID: "f14812ec-6d3c-4957-be87-0454c8c21d65") : secret "dns-default-metrics-tls" not found Oct 31 00:27:51.488437 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488397 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.488385231 +0000 UTC m=+39.754716020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : configmap references non-existent config key: service-ca.crt Oct 31 00:27:51.488498 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488462 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:27:51.488498 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488492 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.488482136 +0000 UTC m=+39.754812922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : secret "router-metrics-certs-default" not found Oct 31 00:27:51.488600 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488545 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:51.488600 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488556 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c65457c94-5cwm4: secret "image-registry-tls" not found Oct 31 00:27:51.488600 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488584 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls podName:8fcf1ba8-6176-418b-8477-6f5b89a16a41 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.488575569 +0000 UTC m=+39.754906356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls") pod "image-registry-7c65457c94-5cwm4" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41") : secret "image-registry-tls" not found Oct 31 00:27:51.488740 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488642 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:51.488740 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488674 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls podName:f4428250-46e8-4f67-bb0c-285938ee2175 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.488664749 +0000 UTC m=+39.754995536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlk72" (UID: "f4428250-46e8-4f67-bb0c-285938ee2175") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:51.488740 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488720 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:27:51.488907 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488747 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls podName:b634720f-79f7-4d34-a971-8d1ba231b3d3 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.48873833 +0000 UTC m=+39.755069116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5h7kl" (UID: "b634720f-79f7-4d34-a971-8d1ba231b3d3") : secret "samples-operator-tls" not found Oct 31 00:27:51.488907 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488816 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:51.488907 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488826 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b88d6f687-f96vv: secret "image-registry-tls" not found Oct 31 00:27:51.488907 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:51.488855 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls podName:c998bbec-e9a8-45b9-8475-50a4e4fb2c35 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.488845853 +0000 UTC m=+39.755176641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls") pod "image-registry-5b88d6f687-f96vv" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35") : secret "image-registry-tls" not found Oct 31 00:27:51.518505 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:51.518453 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" podStartSLOduration=365.167423114 podStartE2EDuration="6m7.51843944s" totalImagesPullingTime="2.351016326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:51.516895439 +0000 UTC m=+35.783226235" watchObservedRunningTime="2026-10-31 00:27:51.51843944 +0000 UTC m=+35.784770236" Oct 31 00:27:53.418515 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.418482 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/0.log" Oct 31 00:27:53.419054 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.418523 2545 generic.go:356] "Generic (PLEG): container finished" podID="341fbe38-f97c-4a00-938b-6c4b9a30197e" containerID="56388cc7525e097a95d8b7e292fbcbd22cdd36487ee05666e48f8cb413e1c4bd" exitCode=255 Oct 31 00:27:53.419054 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.418629 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" event={"ID":"341fbe38-f97c-4a00-938b-6c4b9a30197e","Type":"ContainerDied","Data":"56388cc7525e097a95d8b7e292fbcbd22cdd36487ee05666e48f8cb413e1c4bd"} Oct 31 00:27:53.419170 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.419088 2545 scope.go:122] "RemoveContainer" containerID="56388cc7525e097a95d8b7e292fbcbd22cdd36487ee05666e48f8cb413e1c4bd" Oct 31 00:27:53.421625 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.421199 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" event={"ID":"38807dac-4dc9-4d27-932b-e81189808eb1","Type":"ContainerStarted","Data":"2e73fa5dd63035a3ea152d507ca619a2aab66c1e2f717113b8904fb4b83a68b2"} Oct 31 00:27:53.424718 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.424690 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sbz69" event={"ID":"7c0f42a0-fd35-4121-96b9-1d6363689977","Type":"ContainerStarted","Data":"a3f3b7e65ddb7ac0e3ec5b7e353b778534f5e8e17eea379db3f7cbb1b4118f96"} Oct 31 00:27:53.643894 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.643825 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" podStartSLOduration=365.246972342 podStartE2EDuration="6m9.64380432s" totalImagesPullingTime="4.396831978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:53.643215269 +0000 UTC m=+37.909546065" watchObservedRunningTime="2026-10-31 00:27:53.64380432 +0000 UTC m=+37.910135115" Oct 31 00:27:53.753016 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.752948 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-sbz69" podStartSLOduration=368.481689498 podStartE2EDuration="6m12.752928052s" totalImagesPullingTime="4.271238554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:53.752688495 +0000 UTC m=+38.019019291" watchObservedRunningTime="2026-10-31 00:27:53.752928052 +0000 UTC m=+38.019258849" Oct 31 00:27:53.864167 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.864134 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jd4rk"] Oct 31 00:27:53.868207 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.868178 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:53.917767 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.917713 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:27:53.917934 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.917834 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:27:53.917934 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.917915 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9qgw5\"" Oct 31 00:27:53.922452 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:53.922402 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jd4rk"] Oct 31 00:27:54.019243 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.019077 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sg8v\" (UniqueName: \"kubernetes.io/projected/a83a01b9-9670-484c-ba66-658f61d5f162-kube-api-access-9sg8v\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.019243 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.019204 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.019491 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.019284 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83a01b9-9670-484c-ba66-658f61d5f162-crio-socket\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.019491 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.019336 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83a01b9-9670-484c-ba66-658f61d5f162-data-volume\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.019491 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.019364 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.086003 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.085968 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bcqhp-55jrh"] Oct 31 00:27:54.089706 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.089668 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.120133 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sg8v\" (UniqueName: \"kubernetes.io/projected/a83a01b9-9670-484c-ba66-658f61d5f162-kube-api-access-9sg8v\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.120232 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.120303 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83a01b9-9670-484c-ba66-658f61d5f162-crio-socket\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.120353 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83a01b9-9670-484c-ba66-658f61d5f162-data-volume\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.120380 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.120481 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.120542 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:54.620521807 +0000 UTC m=+38.886852583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : configmap "kube-rbac-proxy" not found Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.120950 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.120987 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:54.620976224 +0000 UTC m=+38.887306996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : secret "insights-runtime-extractor-tls" not found Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.121118 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83a01b9-9670-484c-ba66-658f61d5f162-crio-socket\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.121607 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.121551 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83a01b9-9670-484c-ba66-658f61d5f162-data-volume\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.145669 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.145637 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bcqhp-55jrh"] Oct 31 00:27:54.178031 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.178004 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sg8v\" (UniqueName: \"kubernetes.io/projected/a83a01b9-9670-484c-ba66-658f61d5f162-kube-api-access-9sg8v\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.221972 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.221918 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.222139 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.222054 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjq6\" (UniqueName: \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.222209 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.222169 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.222475 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.222302 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.323775 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.323686 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.323952 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.323817 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.323952 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.323887 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjq6\" (UniqueName: \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.324424 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.324392 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.324535 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.324429 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.324712 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.324690 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.324833 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.324811 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.382276 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.382211 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjq6\" (UniqueName: \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.405303 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.405255 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:27:54.437485 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.434069 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:27:54.437485 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.434517 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/0.log" Oct 31 00:27:54.437485 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.434556 2545 generic.go:356] "Generic (PLEG): container finished" podID="341fbe38-f97c-4a00-938b-6c4b9a30197e" containerID="befbbec3de1d94bdec332280738b16998f7aec9d2a0cbd2a430ceb2330c90e1b" exitCode=255 Oct 31 00:27:54.437485 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.434668 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" event={"ID":"341fbe38-f97c-4a00-938b-6c4b9a30197e","Type":"ContainerDied","Data":"befbbec3de1d94bdec332280738b16998f7aec9d2a0cbd2a430ceb2330c90e1b"} Oct 31 00:27:54.437485 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.434716 2545 scope.go:122] "RemoveContainer" containerID="56388cc7525e097a95d8b7e292fbcbd22cdd36487ee05666e48f8cb413e1c4bd" Oct 31 00:27:54.437485 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.435348 2545 scope.go:122] "RemoveContainer" containerID="befbbec3de1d94bdec332280738b16998f7aec9d2a0cbd2a430ceb2330c90e1b" Oct 31 00:27:54.437485 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.435529 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlcr7_openshift-console-operator(341fbe38-f97c-4a00-938b-6c4b9a30197e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" podUID="341fbe38-f97c-4a00-938b-6c4b9a30197e" Oct 31 00:27:54.627367 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.627333 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.627538 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.627433 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:54.627616 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.627566 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:27:54.627616 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.627611 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.627597863 +0000 UTC m=+39.893928637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : configmap "kube-rbac-proxy" not found Oct 31 00:27:54.627729 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.627679 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:27:54.627729 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:54.627704 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:55.627697366 +0000 UTC m=+39.894028138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : secret "insights-runtime-extractor-tls" not found Oct 31 00:27:54.782416 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:54.782370 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bcqhp-55jrh"] Oct 31 00:27:55.298675 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.298655 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-427g5"] Oct 31 00:27:55.303405 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.302847 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.335183 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.335146 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-cabundle\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.335183 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.335199 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-kube-api-access-gmr5h\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.335393 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.335287 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-key\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.351762 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.351726 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:27:55.351929 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.351774 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:27:55.351929 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.351798 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:27:55.354432 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.354411 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zhjnq\"" Oct 31 00:27:55.355948 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.355846 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-427g5"] Oct 31 00:27:55.435832 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.435800 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-cabundle\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.436014 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.435863 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-kube-api-access-gmr5h\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.436014 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.435898 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-key\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.436494 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.436466 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-cabundle\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.438820 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.438796 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-key\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.440139 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.440120 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:27:55.440745 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.440476 2545 scope.go:122] "RemoveContainer" containerID="befbbec3de1d94bdec332280738b16998f7aec9d2a0cbd2a430ceb2330c90e1b" Oct 31 00:27:55.440745 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.440677 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlcr7_openshift-console-operator(341fbe38-f97c-4a00-938b-6c4b9a30197e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" podUID="341fbe38-f97c-4a00-938b-6c4b9a30197e" Oct 31 00:27:55.442032 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.442008 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqdmn" event={"ID":"7a345f30-ddf3-478a-81e0-5d50cbf38dc4","Type":"ContainerStarted","Data":"f1b58f0206f7efbd9b6de1d24a1a1c0262a7523c5e2aa1a57c32150097a0202b"} Oct 31 00:27:55.443696 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.443673 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" event={"ID":"604f57d5-12dc-4ade-895d-c876688423fb","Type":"ContainerStarted","Data":"15909e726f24571054749daffeee1af025c530a03ced4d10c8c161245062d74a"} Oct 31 00:27:55.445456 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.445428 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" event={"ID":"942dd14b-9b63-4c8a-8362-218b9043307c","Type":"ContainerStarted","Data":"22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2"} Oct 31 00:27:55.445545 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.445468 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" event={"ID":"942dd14b-9b63-4c8a-8362-218b9043307c","Type":"ContainerStarted","Data":"c361feb9917e45dab6a40d71540b1bc98955f2846ed37b03787dad269bd5e4a4"} Oct 31 00:27:55.447340 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.447316 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" event={"ID":"c1caab5f-35c5-4f4d-8f81-0d18527a7ef1","Type":"ContainerStarted","Data":"4c80de115ad061959eeb7cc6e520d0d15112760147081c4405a4890d93892a7b"} Oct 31 00:27:55.452477 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.452460 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:27:55.494460 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.494432 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-kube-api-access-gmr5h\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.536760 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.536723 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:27:55.536947 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.536865 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:27:55.536947 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.536928 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls podName:f14812ec-6d3c-4957-be87-0454c8c21d65 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.536914505 +0000 UTC m=+47.803245277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls") pod "dns-default-z4v42" (UID: "f14812ec-6d3c-4957-be87-0454c8c21d65") : secret "dns-default-metrics-tls" not found Oct 31 00:27:55.537058 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.536977 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:55.537104 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.537091 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:27:55.537149 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.537125 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:27:55.537238 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537212 2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:27:55.537238 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.537233 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:27:55.537409 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537261 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.537246835 +0000 UTC m=+47.803577624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : secret "router-metrics-certs-default" not found Oct 31 00:27:55.537409 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.537321 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:27:55.537409 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.537377 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:27:55.537562 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.537458 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:27:55.537562 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537530 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle podName:a8643bdf-eeca-4201-8d47-e4b1b69eefa2 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.53751484 +0000 UTC m=+47.803845630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle") pod "router-default-5cfd77cbdf-kcvgr" (UID: "a8643bdf-eeca-4201-8d47-e4b1b69eefa2") : configmap references non-existent config key: service-ca.crt Oct 31 00:27:55.537663 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537588 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:55.537663 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537601 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c65457c94-5cwm4: secret "image-registry-tls" not found Oct 31 00:27:55.537967 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537875 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:27:55.537967 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537896 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b88d6f687-f96vv: secret "image-registry-tls" not found Oct 31 00:27:55.537967 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537910 2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:55.537967 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537938 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls podName:c998bbec-e9a8-45b9-8475-50a4e4fb2c35 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.537923793 +0000 UTC m=+47.804254583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls") pod "image-registry-5b88d6f687-f96vv" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35") : secret "image-registry-tls" not found Oct 31 00:27:55.538206 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.537980 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls podName:f4428250-46e8-4f67-bb0c-285938ee2175 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.537962646 +0000 UTC m=+47.804293435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlk72" (UID: "f4428250-46e8-4f67-bb0c-285938ee2175") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:27:55.538206 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.538035 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls podName:8fcf1ba8-6176-418b-8477-6f5b89a16a41 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.538016523 +0000 UTC m=+47.804347303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls") pod "image-registry-7c65457c94-5cwm4" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41") : secret "image-registry-tls" not found Oct 31 00:27:55.538206 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.538060 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:27:55.538206 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.538068 2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:27:55.538206 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.538109 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert podName:48eb4acb-8de7-4b4e-9fea-ca7bf8556014 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.53809888 +0000 UTC m=+47.804429656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4w8gv" (UID: "48eb4acb-8de7-4b4e-9fea-ca7bf8556014") : secret "networking-console-plugin-cert" not found Oct 31 00:27:55.538206 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.538148 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls podName:b634720f-79f7-4d34-a971-8d1ba231b3d3 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:03.538131225 +0000 UTC m=+47.804462037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5h7kl" (UID: "b634720f-79f7-4d34-a971-8d1ba231b3d3") : secret "samples-operator-tls" not found Oct 31 00:27:55.617230 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.617156 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:27:55.639027 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.638990 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:55.639163 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.639124 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:55.639243 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.639226 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:27:55.639314 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.639302 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:57.639283515 +0000 UTC m=+41.905614288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : configmap "kube-rbac-proxy" not found Oct 31 00:27:55.639402 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.639389 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:27:55.639456 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:55.639433 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:57.63942168 +0000 UTC m=+41.905752460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : secret "insights-runtime-extractor-tls" not found Oct 31 00:27:55.659311 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.659254 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" podStartSLOduration=366.877728534 podStartE2EDuration="6m13.659234959s" totalImagesPullingTime="6.781506425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:55.658368812 +0000 UTC m=+39.924699611" watchObservedRunningTime="2026-10-31 00:27:55.659234959 +0000 UTC m=+39.925565754" Oct 31 00:27:55.774058 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.773993 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wqdmn" podStartSLOduration=2.8469539409999998 podStartE2EDuration="8.773976213s" totalImagesPullingTime="5.927022272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:55.772484204 +0000 UTC m=+40.038815002" watchObservedRunningTime="2026-10-31 00:27:55.773976213 +0000 UTC m=+40.040307007" Oct 31 00:27:55.882535 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.882436 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" podStartSLOduration=1.8824185660000001 podStartE2EDuration="1.882418566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:55.8817364 +0000 UTC m=+40.148067209" watchObservedRunningTime="2026-10-31 00:27:55.882418566 +0000 UTC m=+40.148749362" Oct 31 00:27:55.990742 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:55.990684 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" podStartSLOduration=362.500148341 podStartE2EDuration="6m8.990668146s" totalImagesPullingTime="6.490519805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:55.989185216 +0000 UTC m=+40.255516012" watchObservedRunningTime="2026-10-31 00:27:55.990668146 +0000 UTC m=+40.256998941" Oct 31 00:27:56.000799 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:56.000757 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-427g5"] Oct 31 00:27:56.003887 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:27:56.003858 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6a5917_62f5_4f02_b9eb_dcb8ac2a5529.slice/crio-f909c5ab4d06ac2df8ed042c8f12ac297041bf1b1f1c5b15be9924f92b22960b WatchSource:0}: Error finding container f909c5ab4d06ac2df8ed042c8f12ac297041bf1b1f1c5b15be9924f92b22960b: Status 404 returned error can't find the container with id f909c5ab4d06ac2df8ed042c8f12ac297041bf1b1f1c5b15be9924f92b22960b Oct 31 00:27:56.451961 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:56.451918 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8grlc" event={"ID":"a33cb5d1-e145-47db-9c98-2561a4213883","Type":"ContainerStarted","Data":"df4f9feaf102ef92c414c8cb3e45b36874d7d2a599bb5c9f08fce31089e22764"} Oct 31 00:27:56.452422 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:56.452040 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:27:56.453631 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:56.453605 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" event={"ID":"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529","Type":"ContainerStarted","Data":"e0ea4621bdda7d90b536828e2f962be67625f8aee6c6ed2be56f8c0b5d0d900b"} Oct 31 00:27:56.453836 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:56.453644 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" event={"ID":"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529","Type":"ContainerStarted","Data":"f909c5ab4d06ac2df8ed042c8f12ac297041bf1b1f1c5b15be9924f92b22960b"} Oct 31 00:27:56.560296 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:56.559807 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8grlc" podStartSLOduration=33.470620258 podStartE2EDuration="39.559773622s" totalImagesPullingTime="6.089153364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:56.559329502 +0000 UTC m=+40.825660294" watchObservedRunningTime="2026-10-31 00:27:56.559773622 +0000 UTC m=+40.826104418" Oct 31 00:27:56.667582 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:56.667526 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" podStartSLOduration=1.6675133309999999 podStartE2EDuration="1.667513331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:56.666278799 +0000 UTC m=+40.932609597" watchObservedRunningTime="2026-10-31 00:27:56.667513331 +0000 UTC m=+40.933844126" Oct 31 00:27:57.063912 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:57.063882 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-946fm_af99842a-26b2-4e57-aae5-3aceef8f0776/dns-node-resolver/0.log" Oct 31 00:27:57.658966 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:57.658925 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:57.659454 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:57.659058 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:27:57.659454 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:57.659100 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:27:57.659454 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:57.659175 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:01.659156426 +0000 UTC m=+45.925487228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : secret "insights-runtime-extractor-tls" not found Oct 31 00:27:57.659454 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:57.659213 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:27:57.659454 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:57.659264 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:01.659249646 +0000 UTC m=+45.925580434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : configmap "kube-rbac-proxy" not found Oct 31 00:27:58.088730 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:58.088695 2545 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:58.088898 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:58.088755 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:27:58.089109 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:58.089096 2545 scope.go:122] "RemoveContainer" containerID="befbbec3de1d94bdec332280738b16998f7aec9d2a0cbd2a430ceb2330c90e1b" Oct 31 00:27:58.089288 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:27:58.089269 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jlcr7_openshift-console-operator(341fbe38-f97c-4a00-938b-6c4b9a30197e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" podUID="341fbe38-f97c-4a00-938b-6c4b9a30197e" Oct 31 00:27:58.264996 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:58.264967 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fgvdh_d172f407-b9ac-4193-b0f4-fab1c53e6874/node-ca/0.log" Oct 31 00:27:58.864888 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:27:58.864860 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wqdmn_7a345f30-ddf3-478a-81e0-5d50cbf38dc4/serve-healthcheck-canary/0.log" Oct 31 00:28:01.698344 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:01.698306 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:01.698839 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:01.698418 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:01.698839 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:28:01.698541 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:28:01.698839 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:28:01.698635 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls podName:a83a01b9-9670-484c-ba66-658f61d5f162 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:09.698612982 +0000 UTC m=+53.964943779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jd4rk" (UID: "a83a01b9-9670-484c-ba66-658f61d5f162") : secret "insights-runtime-extractor-tls" not found Oct 31 00:28:01.699019 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:01.698886 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:03.617477 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617442 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:03.617477 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617486 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:03.618023 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617503 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:03.618023 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617689 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:03.618023 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617749 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:03.618023 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617834 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:03.618023 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617877 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:28:03.618023 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.617971 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:03.618302 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:28:03.618079 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:28:03.618302 ip-10-0-143-105 kubenswrapper[2545]: E1031 00:28:03.618139 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls podName:f14812ec-6d3c-4957-be87-0454c8c21d65 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:19.618122178 +0000 UTC m=+63.884452953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls") pod "dns-default-z4v42" (UID: "f14812ec-6d3c-4957-be87-0454c8c21d65") : secret "dns-default-metrics-tls" not found Oct 31 00:28:03.618302 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.618255 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:03.620244 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.620216 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:03.620589 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.620559 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:03.620664 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.620574 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:03.620717 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.620682 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:03.621044 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.621027 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:28:03.621307 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.621290 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:03.631320 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.631299 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:03.651456 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.651434 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:03.657517 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.657506 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:28:03.678727 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.678707 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:03.691005 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.690987 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:03.700134 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:03.700115 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:04.152305 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.152279 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c65457c94-5cwm4"] Oct 31 00:28:04.154719 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:04.154687 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcf1ba8_6176_418b_8477_6f5b89a16a41.slice/crio-eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33 WatchSource:0}: Error finding container eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33: Status 404 returned error can't find the container with id eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33 Oct 31 00:28:04.192720 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:04.192685 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8643bdf_eeca_4201_8d47_e4b1b69eefa2.slice/crio-1974af933a7840ab6d3f926fbef7470df6de80d1394695647c1981e1b6429531 WatchSource:0}: Error finding container 1974af933a7840ab6d3f926fbef7470df6de80d1394695647c1981e1b6429531: Status 404 returned error can't find the container with id 1974af933a7840ab6d3f926fbef7470df6de80d1394695647c1981e1b6429531 Oct 31 00:28:04.198831 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.198810 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5cfd77cbdf-kcvgr"] Oct 31 00:28:04.200732 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.200693 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b88d6f687-f96vv"] Oct 31 00:28:04.202259 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:04.202231 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc998bbec_e9a8_45b9_8475_50a4e4fb2c35.slice/crio-3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff WatchSource:0}: Error finding container 3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff: Status 404 returned error can't find the container with id 3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff Oct 31 00:28:04.204544 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:04.204518 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4428250_46e8_4f67_bb0c_285938ee2175.slice/crio-b5bf3124fa94c1795a0b3da7074abe2a8944352b96e58737d2ef1d13ab402056 WatchSource:0}: Error finding container b5bf3124fa94c1795a0b3da7074abe2a8944352b96e58737d2ef1d13ab402056: Status 404 returned error can't find the container with id b5bf3124fa94c1795a0b3da7074abe2a8944352b96e58737d2ef1d13ab402056 Oct 31 00:28:04.245066 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.245033 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72"] Oct 31 00:28:04.247982 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.246146 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl"] Oct 31 00:28:04.248422 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.248342 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4w8gv"] Oct 31 00:28:04.251455 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:04.251433 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48eb4acb_8de7_4b4e_9fea_ca7bf8556014.slice/crio-cca9f767847f4e5c430822b5e4c0ffd4d524069066f42804ba6b82651523f6e2 WatchSource:0}: Error finding container cca9f767847f4e5c430822b5e4c0ffd4d524069066f42804ba6b82651523f6e2: Status 404 returned error can't find the container with id cca9f767847f4e5c430822b5e4c0ffd4d524069066f42804ba6b82651523f6e2 Oct 31 00:28:04.481515 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.481464 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" event={"ID":"8fcf1ba8-6176-418b-8477-6f5b89a16a41","Type":"ContainerStarted","Data":"a83508bb60251551009e491fc2ad8bbba7e1cd662d6d2f99222e59accfe4127b"} Oct 31 00:28:04.481693 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.481545 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" event={"ID":"8fcf1ba8-6176-418b-8477-6f5b89a16a41","Type":"ContainerStarted","Data":"eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33"} Oct 31 00:28:04.481693 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.481592 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:04.482739 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.482714 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" event={"ID":"b634720f-79f7-4d34-a971-8d1ba231b3d3","Type":"ContainerStarted","Data":"139dd654d0ba8d107fd5dec6a7190a22918d33ef81002b33e4d8593f74306cad"} Oct 31 00:28:04.484436 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.484410 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" event={"ID":"a8643bdf-eeca-4201-8d47-e4b1b69eefa2","Type":"ContainerStarted","Data":"0819dc3d462d403b58ae859c825b5323fbe409cf6de1782bd42b38d9c8ab4799"} Oct 31 00:28:04.484540 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.484450 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" event={"ID":"a8643bdf-eeca-4201-8d47-e4b1b69eefa2","Type":"ContainerStarted","Data":"1974af933a7840ab6d3f926fbef7470df6de80d1394695647c1981e1b6429531"} Oct 31 00:28:04.486052 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.486027 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" event={"ID":"c998bbec-e9a8-45b9-8475-50a4e4fb2c35","Type":"ContainerStarted","Data":"de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8"} Oct 31 00:28:04.486178 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.486064 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" event={"ID":"c998bbec-e9a8-45b9-8475-50a4e4fb2c35","Type":"ContainerStarted","Data":"3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff"} Oct 31 00:28:04.486178 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.486145 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:04.487224 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.487199 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" event={"ID":"f4428250-46e8-4f67-bb0c-285938ee2175","Type":"ContainerStarted","Data":"b5bf3124fa94c1795a0b3da7074abe2a8944352b96e58737d2ef1d13ab402056"} Oct 31 00:28:04.488424 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.488400 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" event={"ID":"48eb4acb-8de7-4b4e-9fea-ca7bf8556014","Type":"ContainerStarted","Data":"cca9f767847f4e5c430822b5e4c0ffd4d524069066f42804ba6b82651523f6e2"} Oct 31 00:28:04.597200 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.597130 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" podStartSLOduration=384.597110624 podStartE2EDuration="6m24.597110624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:04.595434036 +0000 UTC m=+48.861764843" watchObservedRunningTime="2026-10-31 00:28:04.597110624 +0000 UTC m=+48.863441425" Oct 31 00:28:04.651771 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.651738 2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:04.654124 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.654096 2545 patch_prober.go:29] interesting pod/router-default-5cfd77cbdf-kcvgr container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Oct 31 00:28:04.654124 ip-10-0-143-105 kubenswrapper[2545]: [+]has-synced ok Oct 31 00:28:04.654124 ip-10-0-143-105 kubenswrapper[2545]: [+]process-running ok Oct 31 00:28:04.654124 ip-10-0-143-105 kubenswrapper[2545]: healthz check failed Oct 31 00:28:04.654325 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.654144 2545 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" podUID="a8643bdf-eeca-4201-8d47-e4b1b69eefa2" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 31 00:28:04.712772 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.712712 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" podStartSLOduration=38.712697576 podStartE2EDuration="38.712697576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:04.711440951 +0000 UTC m=+48.977771747" watchObservedRunningTime="2026-10-31 00:28:04.712697576 +0000 UTC m=+48.979028368" Oct 31 00:28:04.824678 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:04.824585 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" podStartSLOduration=360.824573087 podStartE2EDuration="6m0.824573087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:04.823373873 +0000 UTC m=+49.089704668" watchObservedRunningTime="2026-10-31 00:28:04.824573087 +0000 UTC m=+49.090903882" Oct 31 00:28:05.653121 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:05.653082 2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:06.212112 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.212074 2545 scope.go:122] "RemoveContainer" containerID="befbbec3de1d94bdec332280738b16998f7aec9d2a0cbd2a430ceb2330c90e1b" Oct 31 00:28:06.499107 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.499030 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:28:06.499695 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.499540 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" event={"ID":"341fbe38-f97c-4a00-938b-6c4b9a30197e","Type":"ContainerStarted","Data":"8219c0a65b16b44f94e38c8d5addc46251dcd29c5a7448f24f4b9f0d52778b28"} Oct 31 00:28:06.500285 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.500125 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:06.501086 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.501026 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:06.501550 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.501510 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:06.502163 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.502139 2545 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jlcr7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body= Oct 31 00:28:06.502269 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.502183 2545 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" podUID="341fbe38-f97c-4a00-938b-6c4b9a30197e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused" Oct 31 00:28:06.612835 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:06.612760 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" podStartSLOduration=383.654668037 podStartE2EDuration="6m27.612741118s" totalImagesPullingTime="3.958073081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:06.611387034 +0000 UTC m=+50.877717830" watchObservedRunningTime="2026-10-31 00:28:06.612741118 +0000 UTC m=+50.879071913" Oct 31 00:28:07.509266 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:07.509234 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" event={"ID":"48eb4acb-8de7-4b4e-9fea-ca7bf8556014","Type":"ContainerStarted","Data":"401fd7cbcf102ce62e889cbc84cc6598bb5b3d704b6591d2d5270868c5d69249"} Oct 31 00:28:07.512876 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:07.512854 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:07.619022 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:07.618977 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" podStartSLOduration=337.785533353 podStartE2EDuration="5m40.618963171s" totalImagesPullingTime="2.833429818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:07.617621339 +0000 UTC m=+51.883952134" watchObservedRunningTime="2026-10-31 00:28:07.618963171 +0000 UTC m=+51.885293962" Oct 31 00:28:08.514224 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:08.514181 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" event={"ID":"f4428250-46e8-4f67-bb0c-285938ee2175","Type":"ContainerStarted","Data":"51989fd3b78483e4bde6479dcbb182228af2e8e621e164461a9f8cc2bcbcec22"} Oct 31 00:28:08.516104 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:08.516077 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" event={"ID":"b634720f-79f7-4d34-a971-8d1ba231b3d3","Type":"ContainerStarted","Data":"06f0f0f8cef207e8b77581e7f5e544ad0d43e7a1b3de86bea85c63a49ab6eaa2"} Oct 31 00:28:08.516240 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:08.516126 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" event={"ID":"b634720f-79f7-4d34-a971-8d1ba231b3d3","Type":"ContainerStarted","Data":"0d2c3881a0de11980d520c12ec20b8622943511085f0b888ca71d0564f28a5d0"} Oct 31 00:28:08.624774 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:08.624722 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" podStartSLOduration=386.406275548 podStartE2EDuration="6m29.624707404s" totalImagesPullingTime="3.218431856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:08.622894968 +0000 UTC m=+52.889225762" watchObservedRunningTime="2026-10-31 00:28:08.624707404 +0000 UTC m=+52.891038198" Oct 31 00:28:08.734842 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:08.734765 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" podStartSLOduration=389.586971924 podStartE2EDuration="6m32.734748765s" totalImagesPullingTime="3.147776841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:08.734087655 +0000 UTC m=+53.000418451" watchObservedRunningTime="2026-10-31 00:28:08.734748765 +0000 UTC m=+53.001079559" Oct 31 00:28:09.782374 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:09.782333 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:09.784713 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:09.784687 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:09.787643 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:09.787618 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:10.167600 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:10.167572 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jd4rk"] Oct 31 00:28:10.170404 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:10.170380 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83a01b9_9670_484c_ba66_658f61d5f162.slice/crio-584d682856321d23d7a19f19b0dc15aecd48054e6bc00e802f9eb621e3a8a7ae WatchSource:0}: Error finding container 584d682856321d23d7a19f19b0dc15aecd48054e6bc00e802f9eb621e3a8a7ae: Status 404 returned error can't find the container with id 584d682856321d23d7a19f19b0dc15aecd48054e6bc00e802f9eb621e3a8a7ae Oct 31 00:28:10.524364 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:10.524328 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd4rk" event={"ID":"a83a01b9-9670-484c-ba66-658f61d5f162","Type":"ContainerStarted","Data":"59f47eb4ba8cfdcc2229047b3a50cde47a18e4ca93601f8eea536751ad966428"} Oct 31 00:28:10.524596 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:10.524393 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd4rk" event={"ID":"a83a01b9-9670-484c-ba66-658f61d5f162","Type":"ContainerStarted","Data":"584d682856321d23d7a19f19b0dc15aecd48054e6bc00e802f9eb621e3a8a7ae"} Oct 31 00:28:12.383869 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:12.383775 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:12.533067 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:12.533036 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd4rk" event={"ID":"a83a01b9-9670-484c-ba66-658f61d5f162","Type":"ContainerStarted","Data":"35cb5aabff1a1d1d9d9ab9bc5f899e873142fb0476c390b7636fe9bd24ab4059"} Oct 31 00:28:15.037847 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.037753 2545 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c65457c94-5cwm4"] Oct 31 00:28:15.084332 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.084294 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d"] Oct 31 00:28:15.106677 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.106648 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" Oct 31 00:28:15.152381 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.152304 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d"] Oct 31 00:28:15.155726 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.155700 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:15.156105 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.156080 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:28:15.204927 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.204899 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-85rmc\"" Oct 31 00:28:15.225963 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.225933 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsxx\" (UniqueName: \"kubernetes.io/projected/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e-kube-api-access-ffsxx\") pod \"migrator-66df5d496b-89n4d\" (UID: \"8188efe6-6d2b-46cc-ba95-7782ea3a4e6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" Oct 31 00:28:15.326617 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.326544 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsxx\" (UniqueName: \"kubernetes.io/projected/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e-kube-api-access-ffsxx\") pod \"migrator-66df5d496b-89n4d\" (UID: \"8188efe6-6d2b-46cc-ba95-7782ea3a4e6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" Oct 31 00:28:15.384610 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.384577 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsxx\" (UniqueName: \"kubernetes.io/projected/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e-kube-api-access-ffsxx\") pod \"migrator-66df5d496b-89n4d\" (UID: \"8188efe6-6d2b-46cc-ba95-7782ea3a4e6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" Oct 31 00:28:15.437129 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.437100 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" Oct 31 00:28:15.545820 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.545767 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jd4rk" event={"ID":"a83a01b9-9670-484c-ba66-658f61d5f162","Type":"ContainerStarted","Data":"e5d706b592ec2de31a4668890449a2d1492530a615b2ee8f21eaf750a9c0b7ce"} Oct 31 00:28:15.840694 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.840649 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jd4rk" podStartSLOduration=18.532113643 podStartE2EDuration="22.8406323s" totalImagesPullingTime="4.308518657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:15.660031098 +0000 UTC m=+59.926361892" watchObservedRunningTime="2026-10-31 00:28:15.8406323 +0000 UTC m=+60.106963094" Oct 31 00:28:15.841443 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:15.841419 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d"] Oct 31 00:28:15.843059 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:15.843017 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8188efe6_6d2b_46cc_ba95_7782ea3a4e6e.slice/crio-3cce826f4909ef00a95db549785d5d5f3a06415139943e2fa59de3d2f1eb5a25 WatchSource:0}: Error finding container 3cce826f4909ef00a95db549785d5d5f3a06415139943e2fa59de3d2f1eb5a25: Status 404 returned error can't find the container with id 3cce826f4909ef00a95db549785d5d5f3a06415139943e2fa59de3d2f1eb5a25 Oct 31 00:28:16.551628 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:16.551567 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" event={"ID":"8188efe6-6d2b-46cc-ba95-7782ea3a4e6e","Type":"ContainerStarted","Data":"3cce826f4909ef00a95db549785d5d5f3a06415139943e2fa59de3d2f1eb5a25"} Oct 31 00:28:18.560074 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:18.560034 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" event={"ID":"8188efe6-6d2b-46cc-ba95-7782ea3a4e6e","Type":"ContainerStarted","Data":"4edb629e6befc296ea0d9cf844cdf5c3dd737d099c2dc038a58fc54ddaa4fd42"} Oct 31 00:28:18.674076 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:18.674022 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" podStartSLOduration=1.397288352 podStartE2EDuration="3.674009344s" totalImagesPullingTime="2.276720992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:18.672492795 +0000 UTC m=+62.938823590" watchObservedRunningTime="2026-10-31 00:28:18.674009344 +0000 UTC m=+62.940340139" Oct 31 00:28:19.664458 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:19.664423 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:19.666691 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:19.666669 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:19.971246 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:19.971218 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52f7q\"" Oct 31 00:28:19.974311 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:19.974283 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:20.353921 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:20.353897 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4v42"] Oct 31 00:28:20.356506 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:20.356473 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14812ec_6d3c_4957_be87_0454c8c21d65.slice/crio-99c3425adc5ea75db0dbcb5b0472614400f8963bfb80490ee629088b304c5d52 WatchSource:0}: Error finding container 99c3425adc5ea75db0dbcb5b0472614400f8963bfb80490ee629088b304c5d52: Status 404 returned error can't find the container with id 99c3425adc5ea75db0dbcb5b0472614400f8963bfb80490ee629088b304c5d52 Oct 31 00:28:20.568108 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:20.568074 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4v42" event={"ID":"f14812ec-6d3c-4957-be87-0454c8c21d65","Type":"ContainerStarted","Data":"99c3425adc5ea75db0dbcb5b0472614400f8963bfb80490ee629088b304c5d52"} Oct 31 00:28:21.176965 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.176935 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:21.177352 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.176991 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:28:21.179392 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.179359 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:28:21.224975 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.224947 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:28:21.230072 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.230048 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:21.288960 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.288929 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m2wdr\"" Oct 31 00:28:21.292019 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.292002 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:28:21.372417 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.372392 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dwmlr\"" Oct 31 00:28:21.375480 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.375461 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:21.655917 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.655894 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mf88n"] Oct 31 00:28:21.658936 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:21.658905 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f7bca8_ea23_4b2b_9eaa_b26425121ff5.slice/crio-88456d88d676447b6fb0755c68a06c5e8b6d8ea2638401192324da10cb88ead2 WatchSource:0}: Error finding container 88456d88d676447b6fb0755c68a06c5e8b6d8ea2638401192324da10cb88ead2: Status 404 returned error can't find the container with id 88456d88d676447b6fb0755c68a06c5e8b6d8ea2638401192324da10cb88ead2 Oct 31 00:28:21.751326 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:21.749948 2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ksxrk"] Oct 31 00:28:21.753035 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:21.753002 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09934062_de9f_4e88_b25e_4f4a7947e4ce.slice/crio-4b9b99519ba00a1b4a23a36b03be1fdf331f77c56e50d2ac7f299ccc4f6ade62 WatchSource:0}: Error finding container 4b9b99519ba00a1b4a23a36b03be1fdf331f77c56e50d2ac7f299ccc4f6ade62: Status 404 returned error can't find the container with id 4b9b99519ba00a1b4a23a36b03be1fdf331f77c56e50d2ac7f299ccc4f6ade62 Oct 31 00:28:22.576748 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:22.576715 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mf88n" event={"ID":"77f7bca8-ea23-4b2b-9eaa-b26425121ff5","Type":"ContainerStarted","Data":"88456d88d676447b6fb0755c68a06c5e8b6d8ea2638401192324da10cb88ead2"} Oct 31 00:28:22.578147 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:22.578124 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ksxrk" event={"ID":"09934062-de9f-4e88-b25e-4f4a7947e4ce","Type":"ContainerStarted","Data":"4b9b99519ba00a1b4a23a36b03be1fdf331f77c56e50d2ac7f299ccc4f6ade62"} Oct 31 00:28:23.584509 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.584421 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mf88n" event={"ID":"77f7bca8-ea23-4b2b-9eaa-b26425121ff5","Type":"ContainerStarted","Data":"f1825333987a1e12182e4b2b47e85c1b174c541c34e78aab606d59c3bfc6d31d"} Oct 31 00:28:23.584956 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.584521 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mf88n" event={"ID":"77f7bca8-ea23-4b2b-9eaa-b26425121ff5","Type":"ContainerStarted","Data":"1718f86cd6e1a082af8edef393ae13adb97c57548ad37cd37b8ed8b976b0828b"} Oct 31 00:28:23.586524 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.586496 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4v42" event={"ID":"f14812ec-6d3c-4957-be87-0454c8c21d65","Type":"ContainerStarted","Data":"a4fbe8c5cdf5c85e5a46dbd7dfbfa29434b1c2c8451c7bcd413e21cf11914ab2"} Oct 31 00:28:23.586658 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.586545 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4v42" event={"ID":"f14812ec-6d3c-4957-be87-0454c8c21d65","Type":"ContainerStarted","Data":"61ed2ffe8a204b85309e0bd7d0c79a3e66738fe158ee243a02206220dbc0f9bb"} Oct 31 00:28:23.586658 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.586647 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:23.694373 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.694305 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mf88n" podStartSLOduration=65.118575753 podStartE2EDuration="1m6.694293399s" totalImagesPullingTime="1.575717646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:23.692932697 +0000 UTC m=+67.959263506" watchObservedRunningTime="2026-10-31 00:28:23.694293399 +0000 UTC m=+67.960624196" Oct 31 00:28:23.705633 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.705603 2545 patch_prober.go:29] interesting pod/image-registry-5b88d6f687-f96vv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:28:23.705801 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.705653 2545 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:23.808832 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:23.808610 2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-z4v42" podStartSLOduration=34.62843325 podStartE2EDuration="36.808591251s" totalImagesPullingTime="2.180158001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:23.807635055 +0000 UTC m=+68.073965851" watchObservedRunningTime="2026-10-31 00:28:23.808591251 +0000 UTC m=+68.074922048" Oct 31 00:28:25.045339 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:25.045294 2545 patch_prober.go:29] interesting pod/image-registry-7c65457c94-5cwm4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:28:25.045815 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:25.045354 2545 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" podUID="8fcf1ba8-6176-418b-8477-6f5b89a16a41" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:25.498325 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:25.498280 2545 patch_prober.go:29] interesting pod/image-registry-5b88d6f687-f96vv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:28:25.498503 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:25.498337 2545 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:26.831639 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.831598 2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9wfdm"] Oct 31 00:28:26.834623 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.834595 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.933181 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.933137 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:28:26.933181 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.933168 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:28:26.933420 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.933193 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7hhvp\"" Oct 31 00:28:26.933420 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.933249 2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:28:26.933420 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.933381 2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:28:26.935760 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.935732 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.935888 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.935806 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-tls\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.935941 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.935904 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-metrics-client-ca\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.935941 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.935927 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-textfile\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.936049 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.936026 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.936163 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.936083 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-sys\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.936163 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.936114 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75l4l\" (UniqueName: \"kubernetes.io/projected/be133361-4e22-4617-92fd-610a57bef9e2-kube-api-access-75l4l\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.936163 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.936144 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-wtmp\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:26.936285 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:26.936206 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-root\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037013 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.036977 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037013 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037026 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-sys\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037054 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75l4l\" (UniqueName: \"kubernetes.io/projected/be133361-4e22-4617-92fd-610a57bef9e2-kube-api-access-75l4l\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037078 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-wtmp\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037103 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-root\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037113 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-sys\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037128 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037194 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-tls\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037201 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-root\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037230 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-metrics-client-ca\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037255 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-textfile\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037268 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037268 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-wtmp\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037612 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037586 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-textfile\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.037697 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037668 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.038018 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.037997 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-metrics-client-ca\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.039762 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.039739 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-tls\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.040012 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.039991 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.094683 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.094605 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75l4l\" (UniqueName: \"kubernetes.io/projected/be133361-4e22-4617-92fd-610a57bef9e2-kube-api-access-75l4l\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.150794 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.150741 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:27.163210 ip-10-0-143-105 kubenswrapper[2545]: W1031 00:28:27.163180 2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe133361_4e22_4617_92fd_610a57bef9e2.slice/crio-b3785c7b8f0f4b2633c978829360bcc079fd9e5f5b02bd3c0aa39b015ae4add8 WatchSource:0}: Error finding container b3785c7b8f0f4b2633c978829360bcc079fd9e5f5b02bd3c0aa39b015ae4add8: Status 404 returned error can't find the container with id b3785c7b8f0f4b2633c978829360bcc079fd9e5f5b02bd3c0aa39b015ae4add8 Oct 31 00:28:27.459050 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.459018 2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:28:27.607211 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.607166 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wfdm" event={"ID":"be133361-4e22-4617-92fd-610a57bef9e2","Type":"ContainerStarted","Data":"b3785c7b8f0f4b2633c978829360bcc079fd9e5f5b02bd3c0aa39b015ae4add8"} Oct 31 00:28:27.608923 ip-10-0-143-105 kubenswrapper[2545]: I1031 00:28:27.608894 2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ksxrk" event={"ID":"09934062-de9f-4e88-b25e-4f4a7947e4ce","Type":"ContainerStarted","Data":"866a634a92683fea7d4eabd64bba1da4ef9511984bc763c7ab41953ab6b2c76d"} Oct 31 00:28:27.718843 ip-10-0-143-105 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:28:27.736185 ip-10-0-143-105 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:28:27.736365 ip-10-0-143-105 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:28:27.736416 ip-10-0-143-105 systemd[1]: kubelet.service: Consumed 4.623s CPU time. Oct 31 00:28:27.754321 ip-10-0-143-105 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:28:27.828697 ip-10-0-143-105 kubenswrapper[5890]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:27.828697 ip-10-0-143-105 kubenswrapper[5890]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:28:27.828697 ip-10-0-143-105 kubenswrapper[5890]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:27.828697 ip-10-0-143-105 kubenswrapper[5890]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:27.830833 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830730 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.830833 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830832 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.830833 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830836 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830839 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830842 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830845 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830848 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830851 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830854 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830857 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830860 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830865 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830870 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830873 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830877 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830880 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830883 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830886 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830889 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830891 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830894 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.830933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830897 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830899 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830902 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830905 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830909 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830912 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830915 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830918 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830921 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830923 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830926 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830930 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830933 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830935 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830938 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830941 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830944 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830947 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830950 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830953 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.831420 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830956 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830958 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830961 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830963 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830966 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830970 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830972 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830975 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830978 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830981 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830983 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830986 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830989 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830992 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830994 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.830997 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831001 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831003 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831006 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831009 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.831927 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831011 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831014 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831017 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831019 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831029 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831032 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831034 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831037 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831040 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831042 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831045 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831050 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831054 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831058 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831061 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831064 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831075 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831079 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831082 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.832452 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831085 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831087 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831090 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831093 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831096 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831098 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831101 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831104 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831107 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831110 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831113 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831116 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831118 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831121 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831124 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831126 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831129 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831132 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831141 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831144 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.832944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831146 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831149 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831152 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831154 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831158 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831161 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831949 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831960 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831963 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831966 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831969 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831973 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831975 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831978 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831981 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831984 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831987 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831990 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831992 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.833534 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831995 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.831999 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832001 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832004 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832007 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832010 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832012 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832015 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832018 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832020 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832023 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832026 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832029 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832032 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832035 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832037 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832040 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832042 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832045 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.834211 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832048 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832051 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832054 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832057 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832060 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832062 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832066 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832070 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832073 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832076 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832078 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832081 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832084 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832087 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832091 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832094 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832097 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832100 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832102 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832105 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.835000 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832108 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832111 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832113 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832117 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832119 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832122 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832125 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832128 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832131 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832134 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832137 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832140 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832142 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832146 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832149 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832151 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832154 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832157 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.835666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832161 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832165 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832168 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832171 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832174 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832177 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832179 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832182 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832185 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832188 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832191 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832194 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832196 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832199 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832202 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832204 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832207 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832209 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832212 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832215 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.836184 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832217 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832221 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832223 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832226 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832229 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832231 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832234 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832237 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832240 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832242 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832249 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832252 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832255 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832257 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832260 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832263 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832735 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832740 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832744 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832747 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.836688 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832750 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832753 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832756 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832759 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832762 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832765 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832767 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832770 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832772 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832775 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832778 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832794 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832798 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832803 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832807 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832811 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832815 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832820 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832824 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.837282 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832830 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832835 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832838 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832841 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832845 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832847 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832850 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832853 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832856 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832859 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832861 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832864 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832867 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832870 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832873 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832876 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832879 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832882 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832884 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832887 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.838014 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832890 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832892 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832895 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832898 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832901 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832903 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832906 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832910 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832913 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832916 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832919 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832922 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832925 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832927 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832930 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832933 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832935 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832938 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832941 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.838593 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832945 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832949 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832952 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832959 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832962 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832965 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832967 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832970 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832973 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832975 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832978 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832981 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832983 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832986 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832989 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832992 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832994 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.832997 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833000 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.839105 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833002 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833005 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833022 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833025 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833028 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833031 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833034 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833036 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833039 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833042 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833045 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833048 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833051 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833054 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833056 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833059 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833063 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833066 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833069 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833072 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.839570 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833075 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833077 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833080 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833083 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833086 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833183 5890 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833192 5890 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833200 5890 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833207 5890 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833213 5890 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833217 5890 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833221 5890 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833226 5890 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833229 5890 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833233 5890 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833237 5890 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833240 5890 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833244 5890 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833247 5890 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833250 5890 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833253 5890 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833256 5890 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833259 5890 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:28:27.840090 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833266 5890 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833270 5890 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833273 5890 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833276 5890 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833280 5890 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833286 5890 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833289 5890 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833292 5890 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833296 5890 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833299 5890 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833302 5890 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833307 5890 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833312 5890 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833317 5890 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833323 5890 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833327 5890 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833330 5890 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833333 5890 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833337 5890 flags.go:64] FLAG: --enable-server="true" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833340 5890 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833344 5890 flags.go:64] FLAG: --event-burst="100" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833347 5890 flags.go:64] FLAG: --event-qps="50" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833350 5890 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833354 5890 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:28:27.840648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833357 5890 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833365 5890 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833368 5890 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833372 5890 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833375 5890 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833378 5890 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833381 5890 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833384 5890 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833389 5890 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833393 5890 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833398 5890 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833403 5890 flags.go:64] FLAG: --feature-gates="" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833408 5890 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833413 5890 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833418 5890 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833421 5890 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833424 5890 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833427 5890 flags.go:64] FLAG: --help="false" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833430 5890 flags.go:64] FLAG: --hostname-override="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833434 5890 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833437 5890 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833440 5890 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833443 5890 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:28:27.841305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833447 5890 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833450 5890 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833453 5890 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833456 5890 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833458 5890 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833462 5890 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833465 5890 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833469 5890 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833474 5890 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833479 5890 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833487 5890 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833491 5890 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833494 5890 flags.go:64] FLAG: --lock-file="" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833497 5890 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833500 5890 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833504 5890 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833509 5890 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833512 5890 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833516 5890 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833519 5890 flags.go:64] FLAG: --logging-format="text" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833522 5890 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833525 5890 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833530 5890 flags.go:64] FLAG: --manifest-url="" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833533 5890 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833538 5890 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:28:27.841898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833541 5890 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833545 5890 flags.go:64] FLAG: --max-pods="110" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833550 5890 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833555 5890 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833560 5890 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833565 5890 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833568 5890 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833571 5890 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833574 5890 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833582 5890 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833585 5890 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833588 5890 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833591 5890 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833594 5890 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833597 5890 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833601 5890 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833604 5890 flags.go:64] FLAG: --port="10250" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833609 5890 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833612 5890 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-027a027a8c2c19887" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833616 5890 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833619 5890 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833623 5890 flags.go:64] FLAG: --register-node="true" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833626 5890 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833632 5890 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833637 5890 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833642 5890 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:28:27.842520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833647 5890 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833652 5890 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833655 5890 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833659 5890 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833662 5890 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833665 5890 flags.go:64] FLAG: --runonce="false" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833668 5890 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833671 5890 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833674 5890 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833677 5890 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833680 5890 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833684 5890 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833687 5890 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833690 5890 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833693 5890 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833696 5890 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833699 5890 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833701 5890 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833704 5890 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833707 5890 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833710 5890 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833715 5890 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833718 5890 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833723 5890 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833727 5890 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:28:27.843284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833730 5890 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833735 5890 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833738 5890 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833741 5890 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833744 5890 flags.go:64] FLAG: --v="2" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833748 5890 flags.go:64] FLAG: --version="false" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833753 5890 flags.go:64] FLAG: --vmodule="" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833757 5890 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.833761 5890 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833916 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833920 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833923 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833926 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833929 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833932 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833935 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833938 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833940 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833943 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833946 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833949 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.843998 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833952 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833955 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833958 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833960 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833963 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833966 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833969 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833973 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833976 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833980 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833983 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833986 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833989 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833992 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833995 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.833998 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834001 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834004 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834006 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834009 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.844599 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834012 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834014 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834017 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834020 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834022 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834025 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834028 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834030 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834033 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834036 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834038 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834041 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834044 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834047 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834050 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834052 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834055 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834058 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834061 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834064 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.845303 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834066 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834070 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834073 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834076 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834078 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834081 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834084 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834087 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834090 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834093 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834095 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834098 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834101 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834103 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834106 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834109 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834111 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834114 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834117 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.845857 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834119 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834123 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834126 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834129 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834823 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834856 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834862 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834867 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834872 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834879 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834884 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834889 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834893 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834898 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834903 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834908 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834919 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834923 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834928 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.846384 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834935 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834943 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834948 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834953 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834957 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834962 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834966 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834970 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834974 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834980 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834989 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834993 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.834998 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835004 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835009 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835013 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.846883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.835082 5890 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835219 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835229 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835234 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835240 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835245 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835255 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835261 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835267 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835271 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835276 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835281 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835285 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835290 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835294 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835299 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835304 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835308 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835347 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835467 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.847293 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835477 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835482 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835488 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835493 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835498 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835501 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835504 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835507 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835510 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835513 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835516 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835519 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835522 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835525 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835528 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835531 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835534 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835537 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835540 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835543 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.847944 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835545 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835548 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835551 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835554 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835557 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835559 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835562 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835565 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835568 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835571 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835573 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835578 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835582 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835585 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835588 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835591 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835593 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835597 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835600 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835605 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.848542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835608 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835611 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835613 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835617 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835620 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835623 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835626 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835629 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835632 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835635 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835637 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835640 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835662 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835667 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835671 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835674 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835677 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835680 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835683 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.849071 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835688 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835693 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835697 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835700 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835703 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835706 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835709 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835712 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835715 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835718 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835721 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835724 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835727 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835729 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835732 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835735 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835739 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835742 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835744 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.849547 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835747 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835750 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835753 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835756 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835759 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835761 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835764 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835767 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.835772 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.839855 5890 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.839873 5890 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.839892 5890 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.839897 5890 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839951 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839957 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839961 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839964 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839969 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839974 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.850149 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839977 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839980 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839983 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839986 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839989 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839992 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839995 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.839997 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840001 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840004 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840007 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840009 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840012 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840015 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840019 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840022 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840024 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840027 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840030 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840033 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.850962 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840036 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840039 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840041 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840044 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840047 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840051 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840054 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840056 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840059 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840062 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840064 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840067 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840071 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840073 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840076 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840079 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840082 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840084 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840087 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.851763 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840090 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840093 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840096 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840099 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840101 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840106 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840111 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840115 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840118 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840121 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840123 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840127 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840129 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840132 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840135 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840137 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840140 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840143 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840146 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.852558 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840150 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840153 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840155 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840158 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840161 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840163 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840166 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840169 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840172 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840174 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840177 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840180 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840182 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840185 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840188 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840190 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840193 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840196 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840198 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840201 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.853104 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840204 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840207 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840210 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840212 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840216 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840218 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840221 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840224 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840227 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840229 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840232 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840235 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840238 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840241 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840244 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840247 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840250 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840253 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840255 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840258 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.853591 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840261 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840263 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.840289 5890 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840435 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840440 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840443 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840446 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840449 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840451 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840454 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840457 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840460 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840463 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840466 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840468 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840471 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.854107 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840474 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840477 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840480 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840483 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840486 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840488 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840491 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840494 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840496 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840499 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840502 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840505 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840507 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840510 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840513 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840515 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840518 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840520 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840523 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840526 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.854641 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840529 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840531 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840534 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840537 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840540 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840542 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840545 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840547 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840550 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840553 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840556 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840559 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840561 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840564 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840567 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840570 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840572 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840575 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840578 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840580 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.855167 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840583 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840586 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840589 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840593 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840595 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840598 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840601 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840604 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840606 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840609 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840611 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840614 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840617 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840620 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840623 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840625 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840628 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840632 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840636 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.855677 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840640 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840642 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840645 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840648 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840651 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840654 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840657 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840659 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840662 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840665 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840668 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840671 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840674 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840676 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840680 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840684 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840688 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840691 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840694 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.856681 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840696 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840699 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840702 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840705 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840707 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840710 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840713 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840715 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840718 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840721 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840723 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840726 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840729 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840732 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840735 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.857597 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.840757 5890 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840805 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840809 5890 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840813 5890 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840815 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840818 5890 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840821 5890 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840824 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840827 5890 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840830 5890 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840833 5890 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840836 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840838 5890 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840841 5890 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840844 5890 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840847 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840850 5890 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840853 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840856 5890 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840859 5890 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:27.858266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840861 5890 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840864 5890 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840867 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840869 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840872 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840875 5890 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840877 5890 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840880 5890 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840883 5890 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840886 5890 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840889 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840891 5890 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840896 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840899 5890 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840902 5890 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840905 5890 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840908 5890 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840910 5890 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840913 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840916 5890 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:27.859164 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840919 5890 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840922 5890 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840926 5890 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840929 5890 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840931 5890 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840934 5890 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840937 5890 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840940 5890 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840942 5890 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840946 5890 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840949 5890 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840951 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840954 5890 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840957 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840960 5890 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840963 5890 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840966 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840968 5890 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840971 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840974 5890 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:27.860036 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840976 5890 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840979 5890 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840982 5890 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840985 5890 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840989 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840991 5890 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840994 5890 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.840997 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841001 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841003 5890 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841006 5890 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841010 5890 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841014 5890 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841017 5890 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841025 5890 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841028 5890 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841030 5890 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841033 5890 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841036 5890 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:27.860528 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841038 5890 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841041 5890 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841044 5890 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841047 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841050 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841053 5890 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841055 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841058 5890 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841062 5890 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841066 5890 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841069 5890 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841072 5890 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841075 5890 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841077 5890 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841080 5890 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841083 5890 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841086 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841090 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841093 5890 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:27.861048 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841095 5890 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841098 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841101 5890 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841105 5890 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841108 5890 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841111 5890 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841114 5890 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841116 5890 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:28:27.841120 5890 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.841177 5890 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.842160 5890 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.842229 5890 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.842411 5890 server.go:1014] "Starting client certificate rotation" Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.842557 5890 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.842620 5890 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:22:16 +0000 UTC" deadline="2028-07-08 17:25:54.40683046 +0000 UTC" Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.842668 5890 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14800h57m26.564166032s" Oct 31 00:28:27.861505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.842957 5890 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:28:27.861918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.844247 5890 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:28:27.861918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.845549 5890 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:28:27.861918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.847465 5890 log.go:25] "Validated CRI v1 image API" Oct 31 00:28:27.861918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.848211 5890 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:28:27.861918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.852051 5890 fs.go:135] Filesystem UUIDs: map[03dcec63-aa92-47fd-a621-76cd90240bf1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9b27e265-8066-4aa4-816d-8f33486942be:/dev/nvme0n1p4] Oct 31 00:28:27.862088 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.852077 5890 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a4ff567ada319cd933ca254ed30e29c278c7c6bbb3367587b5905b3f8e22b23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a4ff567ada319cd933ca254ed30e29c278c7c6bbb3367587b5905b3f8e22b23/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d77d2c8b3f59aca5189494c2aae27361a48344f3aca5896afe11a935a00b979/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d77d2c8b3f59aca5189494c2aae27361a48344f3aca5896afe11a935a00b979/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/139dd654d0ba8d107fd5dec6a7190a22918d33ef81002b33e4d8593f74306cad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/139dd654d0ba8d107fd5dec6a7190a22918d33ef81002b33e4d8593f74306cad/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1974af933a7840ab6d3f926fbef7470df6de80d1394695647c1981e1b6429531/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1974af933a7840ab6d3f926fbef7470df6de80d1394695647c1981e1b6429531/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21e4e1e7dab09e551ef7ef8b3a6d1c8b0a0de1a66aa69ce85ea84b2ec9237b8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21e4e1e7dab09e551ef7ef8b3a6d1c8b0a0de1a66aa69ce85ea84b2ec9237b8c/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/238ed5358e983b13e1f4e6c70c23cd89186d0980ed12c716297a0c2227b78555/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/238ed5358e983b13e1f4e6c70c23cd89186d0980ed12c716297a0c2227b78555/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cce826f4909ef00a95db549785d5d5f3a06415139943e2fa59de3d2f1eb5a25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cce826f4909ef00a95db549785d5d5f3a06415139943e2fa59de3d2f1eb5a25/userdata/shm major:0 minor:459 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/455f84d7c40eb10d34fac51cc723697c75c82cc6aee4d565cd06aedb57d679bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/455f84d7c40eb10d34fac51cc723697c75c82cc6aee4d565cd06aedb57d679bd/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b9b99519ba00a1b4a23a36b03be1fdf331f77c56e50d2ac7f299ccc4f6ade62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b9b99519ba00a1b4a23a36b03be1fdf331f77c56e50d2ac7f299ccc4f6ade62/userdata/shm major:0 minor:478 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/562095794d42c600a66f4491df56ef216cf5866a1b6c05240d0ea566189371d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/562095794d42c600a66f4491df56ef216cf5866a1b6c05240d0ea566189371d7/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/584d682856321d23d7a19f19b0dc15aecd48054e6bc00e802f9eb621e3a8a7ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/584d682856321d23d7a19f19b0dc15aecd48054e6bc00e802f9eb621e3a8a7ae/userdata/shm major:0 minor:438 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f9fbd2fafd64df2cfe50c61ce867568b5fd152e6b73eaef39ff959c3bca5fc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f9fbd2fafd64df2cfe50c61ce867568b5fd152e6b73eaef39ff959c3bca5fc8/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/644ef22a9afc953fb3f4704023c5a386721cfee1fe596f82893e04f254f75168/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/644ef22a9afc953fb3f4704023c5a386721cfee1fe596f82893e04f254f75168/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/787eb665916078990674eb157d806f8a99ddbaf0b5ab0ac68d6556b7a25d91a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/787eb665916078990674eb157d806f8a99ddbaf0b5ab0ac68d6556b7a25d91a6/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8607ad7be1f639a72522fe3256f13461fc5e4c23e3f1a660e91809b30f61ce9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8607ad7be1f639a72522fe3256f13461fc5e4c23e3f1a660e91809b30f61ce9c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88456d88d676447b6fb0755c68a06c5e8b6d8ea2638401192324da10cb88ead2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88456d88d676447b6fb0755c68a06c5e8b6d8ea2638401192324da10cb88ead2/userdata/shm major:0 minor:476 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98bdb3de1e11569289361a6f2d17a2dd9c2753abafbf9349abbd09c33ff010a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98bdb3de1e11569289361a6f2d17a2dd9c2753abafbf9349abbd09c33ff010a7/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99c3425adc5ea75db0dbcb5b0472614400f8963bfb80490ee629088b304c5d52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99c3425adc5ea75db0dbcb5b0472614400f8963bfb80490ee629088b304c5d52/userdata/shm major:0 minor:470 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1e48c120d9af5509cb12c28d135e33e479156741fcbd392956ae5bab771233f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1e48c120d9af5509cb12c28d135e33e479156741fcbd392956ae5bab771233f/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3785c7b8f0f4b2633c978829360bcc079fd9e5f5b02bd3c0aa39b015ae4add8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3785c7b8f0f4b2633c978829360bcc079fd9e5f5b02bd3c0aa39b015ae4add8/userdata/shm major:0 minor:509 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5bf3124fa94c1795a0b3da7074abe2a8944352b96e58737d2ef1d13ab402056/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5bf3124fa94c1795a0b3da7074abe2a8944352b96e58737d2ef1d13ab402056/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb593b0969779537fd541df460e37740c48c8dbbf3f77bcb14c77cac2cf76046/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb593b0969779537fd541df460e37740c48c8dbbf3f77bcb14c77cac2cf76046/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c361feb9917e45dab6a40d71540b1bc98955f2846ed37b03787dad269bd5e4a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c361feb9917e45dab6a40d71540b1bc98955f2846ed37b03787dad269bd5e4a4/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cca9f767847f4e5c430822b5e4c0ffd4d524069066f42804ba6b82651523f6e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cca9f767847f4e5c430822b5e4c0ffd4d524069066f42804ba6b82651523f6e2/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ceda5fc6e9c6871ca5ca5788ad24b6b065aef2b7ac53bdc9f4763314b2cecf15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ceda5fc6e9c6871ca5ca5788ad24b6b065aef2b7ac53bdc9f4763314b2cecf15/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1fb504d1880f2e5fdc161ec56d44e5306c094421375ef07005ddd5b036f46ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1fb504d1880f2e5fdc161ec56d44e5306c094421375ef07005ddd5b036f46ff/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8b4d5b4eaa16a4737e886726db99da7327961d4b9d5dd0533cdc53f4e5f3f70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8b4d5b4eaa16a4737e886726db99da7327961d4b9d5dd0533cdc53f4e5f3f70/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e93f8991e164e93bf181abdcaa43c1414300a76d57dc3bc293eb358881a8b188/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e93f8991e164e93bf181abdcaa43c1414300a76d57dc3bc293eb358881a8b188/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f67baedb4acce39a4212f5fdc7726fbcd15f0ab2cc8b6b5ff968137c8486a995/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f67baedb4acce39a4212f5fdc7726fbcd15f0ab2cc8b6b5ff968137c8486a995/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f909c5ab4d06ac2df8ed042c8f12ac297041bf1b1f1c5b15be9924f92b22960b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f909c5ab4d06ac2df8ed042c8f12ac297041bf1b1f1c5b15be9924f92b22960b/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa58d08bd48ddd010c2eac4a2ec4af8f47902b4cc1186e99823eef9be713fd96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa58d08bd48ddd010c2eac4a2ec4af8f47902b4cc1186e99823eef9be713fd96/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2/volumes/kubernetes.io~projected/kube-api-access-mjcmw:{mountpoint:/var/lib/kubelet/pods/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2/volumes/kubernetes.io~projected/kube-api-access-mjcmw major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09934062-de9f-4e88-b25e-4f4a7947e4ce/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/09934062-de9f-4e88-b25e-4f4a7947e4ce/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:475 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~projected/kube-api-access-wf7dm:{mountpoint:/var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~projected/kube-api-access-wf7dm major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/341fbe38-f97c-4a00-938b-6c4b9a30197e/volumes/kubernetes.io~projected/kube-api-access-kjb6d:{mountpoint:/var/lib/kubelet/pods/341fbe38-f97c-4a00-938b-6c4b9a30197e/volumes/kubernetes.io~projected/kube-api-access-kjb6d major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/341fbe38-f97c-4a00-938b-6c4b9a30197e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/341fbe38-f97c-4a00-938b-6c4b9a30197e/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38807dac-4dc9-4d27-932b-e81189808eb1/volumes/kubernetes.io~projected/kube-api-access-lkjvm:{mountpoint:/var/lib/kubelet/pods/38807dac-4dc9-4d27-932b-e81189808eb1/volumes/kubernetes.io~projected/kube-api-access-lkjvm major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38807dac-4dc9-4d27-932b-e81189808eb1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/38807dac-4dc9-4d27-932b-e81189808eb1/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48eb4acb-8de7-4b4e-9fea-ca7bf8556014/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/48eb4acb-8de7-4b4e-9fea-ca7bf8556014/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529/volumes/kubernetes.io~projected/kube-api-access-gmr5h:{mountpoint:/var/lib/kubelet/pods/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529/volumes/kubernetes.io~projected/kube-api-access-gmr5h major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529/volumes/kubernetes.io~secret/signing-key major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53a51104-b6b6-4c74-a3c0-e2c5c378f28f/volumes/kubernetes.io~projected/kube-api-access-9sz7n:{mountpoint:/var/lib/kubelet/pods/53a51104-b6b6-4c74-a3c0-e2c5c378f28f/volumes/kubernetes.io~projected/kube-api-access-9sz7n major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604f57d5-12dc-4ade-895d-c876688423fb/volumes/kubernetes.io~projected/kube-api-access-5pfnd:{mountpoint:/var/lib/kubelet/pods/604f57d5-12dc-4ade-895d-c876688423fb/volumes/kubernetes.io~projected/kube-api-access-5pfnd major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604f57d5-12dc-4ade-895d-c876688423fb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/604f57d5-12dc-4ade-895d-c876688423fb/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77f7bca8-ea23-4b2b-9eaa-b26425121ff5/volumes/kubernetes.io~projected/kube-api-access-9f48r:{mountpoint:/var/lib/kubelet/pods/77f7bca8-ea23-4b2b-9eaa-b26425121ff5/volumes/kubernetes.io~projected/kube-api-access-9f48r major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77f7bca8-ea23-4b2b-9eaa-b26425121ff5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/77f7bca8-ea23-4b2b-9eaa-b26425121ff5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volumes/kubernetes.io~projected/kube-api-access-jnwkw:{mountpoint:/var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volumes/kubernetes.io~projected/kube-api-access-jnwkw major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79c0a7af-0b6e-4056-8e2d-166885cb98dc/volumes/kubernetes.io~projected/kube-api-access-pkwx2:{mountpoint:/var/lib/kubelet/pods/79c0a7af-0b6e-4056-8e2d-166885cb98dc/volumes/kubernetes.io~projected/kube-api-access-pkwx2 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a345f30-ddf3-478a-81e0-5d50cbf38dc4/volumes/kubernetes.io~projected/kube-api-access-4fz89:{mountpoint:/var/lib/kubelet/pods/7a345f30-ddf3-478a-81e0-5d50cbf38dc4/volumes/kubernetes.io~projected/kube-api-access-4fz89 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a345f30-ddf3-478a-81e0-5d50cbf38dc4/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7a345f30-ddf3-478a-81e0-5d50cbf38dc4/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0f42a0-fd35-4121-96b9-1d6363689977/volumes/kubernetes.io~projected/kube-api-access-lhqq5:{mountpoint:/var/lib/kubelet/pods/7c0f42a0-fd35-4121-96b9-1d6363689977/volumes/kubernetes.io~projected/kube-api-access-lhqq5 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0f42a0-fd35-4121-96b9-1d6363689977/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7c0f42a0-fd35-4121-96b9-1d6363689977/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e/volumes/kubernetes.io~projected/kube-api-access-ffsxx:{mountpoint:/var/lib/kubelet/pods/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e/volumes/kubernetes.io~projected/kube-api-access-ffsxx major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/kube-api-access-4rwh6:{mountpoint:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/kube-api-access-4rwh6 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/registry-tls major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9316842b-0133-456e-abeb-e0b38e7f04ff/volumes/kubernetes.io~projected/kube-api-access-hjtjc:{mountpoint:/var/lib/kubelet/pods/9316842b-0133-456e-abeb-e0b38e7f04ff/volumes/kubernetes.io~projected/kube-api-access-hjtjc major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/942dd14b-9b63-4c8a-8362-218b9043307c/volumes/kubernetes.io~projected/kube-api-access-rvjq6:{mountpoint:/var/lib/kubelet/pods/942dd14b-9b63-4c8a-8362-218b9043307c/volumes/kubernetes.io~projected/kube-api-access-rvjq6 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a33cb5d1-e145-47db-9c98-2561a4213883/volumes/kubernetes.io~projected/kube-api-access-t8q8z:{mountpoint:/var/lib/kubelet/pods/a33cb5d1-e145-47db-9c98-2561a4213883/volumes/kubernetes.io~projected/kube-api-access-t8q8z major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4090b54-5b6a-49f9-b092-d7c309f3d913/volumes/kubernetes.io~projected/kube-api-access-f88lv:{mountpoint:/var/lib/kubelet/pods/a4090b54-5b6a-49f9-b092-d7c309f3d913/volumes/kubernetes.io~projected/kube-api-access-f88lv major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83a01b9-9670-484c-ba66-658f61d5f162/volumes/kubernetes.io~projected/kube-api-access-9sg8v:{mountpoint:/var/lib/kubelet/pods/a83a01b9-9670-484c-ba66-658f61d5f162/volumes/kubernetes.io~projected/kube-api-access-9sg8v major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83a01b9-9670-484c-ba66-658f61d5f162/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a83a01b9-9670-484c-ba66-658f61d5f162/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~projected/kube-api-access-8rpfb:{mountpoint:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~projected/kube-api-access-8rpfb major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/metrics-certs major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae509a6d-ba84-42c0-9a51-0e7971cb9d68/volumes/kubernetes.io~projected/kube-api-access-csgw4:{mountpoint:/var/lib/kubelet/pods/ae509a6d-ba84-42c0-9a51-0e7971cb9d68/volumes/kubernetes.io~projected/kube-api-access-csgw4 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af99842a-26b2-4e57-aae5-3aceef8f0776/volumes/kubernetes.io~projected/kube-api-access-54zw2:{mountpoint:/var/lib/kubelet/pods/af99842a-26b2-4e57-aae5-3aceef8f0776/volumes/kubernetes.io~projected/kube-api-access-54zw2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b634720f-79f7-4d34-a971-8d1ba231b3d3/volumes/kubernetes.io~projected/kube-api-access-m5tzb:{mountpoint:/var/lib/kubelet/pods/b634720f-79f7-4d34-a971-8d1ba231b3d3/volumes/kubernetes.io~projected/kube-api-access-m5tzb major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b634720f-79f7-4d34-a971-8d1ba231b3d3/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/b634720f-79f7-4d34-a971-8d1ba231b3d3/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~projected/kube-api-access-75l4l:{mountpoint:/var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~projected/kube-api-access-75l4l major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:507 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:503 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1/volumes/kubernetes.io~projected/kube-api-access-cwpcz:{mountpoint:/var/lib/kubelet/pods/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1/volumes/kubernetes.io~projected/kube-api-access-cwpcz major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c63af1f1-ad79-4694-955a-25eb509899d7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c63af1f1-ad79-4694-955a-25eb509899d7/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8057563-4fad-49a2-8d82-4902eca5fa57/volumes/kubernetes.io~projected/kube-api-access-g2h76:{mountpoint:/var/lib/kubelet/pods/c8057563-4fad-49a2-8d82-4902eca5fa57/volumes/kubernetes.io~projected/kube-api-access-g2h76 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/kube-api-access-clwt9:{mountpoint:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/kube-api-access-clwt9 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/registry-tls major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172f407-b9ac-4193-b0f4-fab1c53e6874/volumes/kubernetes.io~projected/kube-api-access-tf86f:{mountpoint:/var/lib/kubelet/pods/d172f407-b9ac-4193-b0f4-fab1c53e6874/volumes/kubernetes.io~projected/kube-api-access-tf86f major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14812ec-6d3c-4957-be87-0454c8c21d65/volumes/kubernetes.io~projected/kube-api-access-g5tkt:{mountpoint:/var/lib/kubelet/pods/f14812ec-6d3c-4957-be87-0454c8c21d65/volumes/kubernetes.io~projected/kube-api-access-g5tkt major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14812ec-6d3c-4957-be87-0454c8c21d65/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f14812ec-6d3c-4957-be87-0454c8c21d65/volumes/kubernetes.io~secret/metrics-tls major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4428250-46e8-4f67-bb0c-285938ee2175/volumes/kubernetes.io~projected/kube-api-access-f8jxj:{mountpoint:/var/lib/kubelet/pods/f4428250-46e8-4f67-bb0c-285938ee2175/volumes/kubernetes.io~projected/kube-api-access-f8jxj major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4428250-46e8-4f67-bb0c-285938ee2175/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/f4428250-46e8-4f67-bb0c-285938ee2175/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fee32755-d814-417f-8311-6a1dc5b60fa5/volumes/kubernetes.io~projected/kube-api-access-rj77k:{mountpoint:/var/lib/kubelet/pods/fee32755-d814-417f-8311-6a1dc5b60fa5/volumes/kubernetes.io~projected/kube-api-access-rj77k major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ca29f357eb703b76cd82d57dfcd55200616fbed863f0e335dd0b22273dacafdd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/649c845b45359abf91fc6d965f3b18badcd4ef076f17688e62c3eb5380718e74/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ae8fb8a875047a36008a1d071522c7345d9c49514001ac92aa1f5bf2d2c3e53b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8ea3dcfd77ad226483632033dec7271e7c8c6502699a223c8a2fc552ef544b81/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/db609121a3225dd546a01b351ec66dc62e25cfaded0f10e201b859e901ba4c6e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/31a9479367172f1c771c974600abd83c85363033835a00f2b93bd5288ba8412f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fcc106b2722c033051d73d39e89ebf7e32f389008b441431dce9ddceab5026ab/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3893c9470de306485826664ab87a4a837582e0a7562e45d153fd27c7e738e13e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b887793b4bf001cb695feea3be03b34a4f846c2c0fe41566a8e41046f219cb30/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/51ab64630291edf24298fa679fb1263446209aac5cc097121c88858e5cf1d60d/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/57963bf06a8d56dce02c5e5cf466222648d77076b0338cd9e4b016d04001bf6f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/a7f9d5a67d43053941c11979558ecae54c5e21d5334397a6e1bec21ffe04b6ee/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/793927c1c88ceadecd4f6faceba0216265a40e391397b2820feb92168c9bbc3e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b39092a7b46b80ca073865bf8999ffed9b0a86073adddd0d43160ab67d962ebb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c7792656e013666fa2b590c2e0c6980851b43eeac6ecf7c4c1f834b8b0b59a3c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1633f1730a514f33d4af386222f9af9f5f956adefe17e0ccc57b4df8b9043b78/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f9733b4e06e8d8690864047dc71859fd610887bbf6661a65f5ff1cfc72fdf685/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/07af1b0329cd059fddc1661c1c14036ef13ba8fbceec0c8cb40adc722552bcdb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/535755e2291f3563531c8e6cf82b8989547852ea25c2082e1782b92c95adf5de/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/38389050c6351a31a6ef947fd6ba851d41fcd590d5245adbd53618a7f1d77f90/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/740a3dd5da30ba458e01d67b8f50ac4c1d3ed95af6b942a40f70022b5881cf07/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b750db72153b5df79e40a454dddff24c5a2d3e8aca3abbc23c431a7c5977e842/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/ee778db3801c1eb2718e2f33385ffcc888446bc398564e2e7fae44795d00fd72/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a1ab4f4ab6575948a535e69553766bdc90ff17aa0bbe0c6416c1fdc456733f45/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a316d94bebc2c338f289e5718ad7940bd7230d033b726e58c9869b01b96a593f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/85663a8a0608b876f28e3efd9d63eecd99281581ee3c542f99fcb2391690539e/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/629bf9d5c24adf98d65c58502ede26e735ea1fdeef3d335572aa23d58c62546b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/413eaeb974e255c734337f4e97f564b49cf729914601db9f34aa8c83a45de536/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/97520c802fb5cc2cc4e37c4a5ae30d44c68cd34ff8fda4ee6350bd1a07e9a86c/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/fd115e3ead4a671c84847868a9b9129675d11f32c0ee6309b5849eeac1981535/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/2e9420924901a3c1c691366a11abd3367e810b84160dd8e3885e131168e3cf7b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/692776df5bc6f19ebfa75db00e16acdad99c0949cdb706e76984f56d4229ee08/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/1fb4ed352a114778b956909d691d3f8226368307b9d7c0c097c46ce240c3e0b8/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/894e3a296cfb40d7952318961b2e336a5bbe90f34236e094fa0af88f3faf5f80/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/162e3cf19011ef9bbb3e9d4b7a23ddd4a728729c5b7ace67d501aa9d247980b8/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/b6a03718cfaf54242662b432fe43aed7fbd1433aa0f9882e5efc7be4ff593e02/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/887e7e84da2befee240e2e814a0618a65d3f61fe70252fe8d0dd016714d0e9dc/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/71a921b7eb48243ed4558cb1307cac9f910b670ce70e0fb1ea320881db06ad0c/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/70a0a06967b8f66d02369e958ae91c031a8dc78906ddfa256f135ec1fc722488/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/3f89e3fd8d3ad75cc8596a11f3a8e3821537b98116413d1a5a2e04201e0a7583/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/b98f24586ed6cbe9a33b1d4cacc368ca8872ab520b8b9d842e59a69af900afe0/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/495b10b01cfefc7070a31ab2c5f55a869e4b42c12f3cab8de9e6c34265e8c805/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/4d77c9ec906d1f6b71e2b422c58d4ffad00fcbb9b5dabeaa6562ae6f6e58b02d/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/5e7dd6e6ad3636fec2bd18e6e5322251db1ab7ed64c948c4ca26cb810bb2b487/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/82c155c0a2275ce819eaae35fcc57feac91ef5eccc4a324093979bff4d6ae06e/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/7e88d26c177abd0afd6324191bf9bf645def02446002736b99270a0a7faa1116/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/ca6b267cedd3fbd6f851fbe38ccf11078ca80a41dc7a46cd7059cfb15e7eef18/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/87a991c14c39379fda79eb8b6834c8551e80e63adfafec367bfc8731063d6114/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/0e60724615e9262153d80ee4dcf46c160de7c8f6b9376a52aff921dc74846667/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/3f5602274280db1c0b356fdc26d3dabf1545c88e61f5897048522bd2aa14c379/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/415069eac5d88cf99b709c7324758f7e99de9c6ea2fda541c4c63db5d63f2666/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/29f2f081d57a97aec88f5bde2d815db534b94b2a4451563ca25592c569a73f11/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/fb5d133ce376516f2839a53d58c996996a5b77a7df40dfbff3ed18547fceecd8/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/2f838843ad904a098998893e0e1b8a8c9478ffa36fccb5efd1dc9a032b8e1a4b/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/f253ffc2f966e166ed98cbc0b01eb5327f912c6bd016e439d76d4906fb3dba78/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/a2b6f93121a9be0b48745dfb6d79bc5272ae8c892e1c60a66e39f67e2b7132ff/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/8ca602d5f063c6d6c4c5c1d7d9ca4c6d7c83867f67e52b2545536e9a9671ed00/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/383fe569378f9316cdd9e7436ec4717b81e787a428ff06afb10b67b0e4f4abc0/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/e34efd27769725b98f28d7984e25ab246caf112389505fa7daca9553d91fb0df/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/889a8d6e1418263b3fc0b102ec529655e9c10c273e06eb2aa4fb1424a3a4f917/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/0ca28b613702f20a84f383b06e79ac460200419e785c51b9b449bd522a6f2dd0/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/b72c96a2e8fc82a448636e9387aa233d8d39402840a3183c376984033c41768f/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/c0cd1e9929dd5444b20262893824751285dd0229c90ae627b4d1b66402ed6e92/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/3c62fea844c2c838a377c40b866d137ab3ac721d8d1eb9e80bc4bc722b926b35/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/964038bb51130c15d99ae63c2d6668e75189d08c4d4d4b276388eac4aa0388d9/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/2d68d391513d7f17b0cef017e8322ff556b58d58f3f967f4d3bb5b8d87703426/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/eae1cb53bd46a80691ec7d22875a87893c18e8eae731aba687452238b137893e/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/8fb083512dcedc86e153cb351219407ef919586875f0681a9b269d48d8cd462f/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/956e4d58ecff59371f80d3e7fc8e6c1290fef4ebf37b3a4f03dd4b597eafbeaf/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/247c1c0e3511680ac128d439ccc045f2e6f705213df3141389763f8fcd881dd6/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/9c9d75a5e1bafb6a8e2fcc4e3377ccd0da540a7d195cb9bda41ca9d01fd94473/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-492:{mountpoint:/var/lib/containers/storage/overlay/3811d7a832b2e80dd7bcda54ba850392e14a45d6c9fa28911b83c3933df9291b/merged major:0 minor:492 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/67a888c3a2801ae2eb148fdc95a8b2514866a38d2b6cdd880ba28237c2852e7b/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-511:{mountpoint:/var/lib/containers/storage/overlay/542a2f9f073588aa315cdb7bdd5badf563af1b001058ea145714255cc8524982/merged major:0 minor:511 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/d36214efd7886b23a6e6c42c7e2a9d4d3900a32c4c43ec6d18c8ff0a7ad2e1a6/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/51c2e34608fbfc0e699dd0de1552412f926865887508198c5f17e9da024e8902/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/22de530fdcbd62ab04b93081d96ae5d2f1c13644aac8a4d96c9ad131f3f2cda8/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/449ca5e464e6681e445febc75be50606d00b59c7de0875d5ad774d45126d5be3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/210d7de442c818933e14b6ff845e3c4046c222ba0db164a275b015220fcfd203/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/752a84ec65ca2fccd08c09931e32448e5102997338a6f20855c9daa6c31734bc/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ec36184cf6ca64eb4f98ffa8bda7de969f48af25c0d891f15c1593e4458cd770/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6eaeb5f229ebe86cd1c45ffaa28053ff28daa718ddad5169f8321a794dbe336b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5e534d2465f62927a7cdb0d272d056041294d2f2335387078e9eb475d8b9260c/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c4ff9cc7c1fa8a304ae49298e59b07d666671c7b881c03b18385ae7ddc85706e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/207d201d69fc311982d81f43b504ffb9a33f78581d82a2074d233e8d1a93073f/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 31 00:28:27.862776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.862224 5890 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:27.861731384 +0000 UTC m=+0.063981808 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101681 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29f4aee6f889f5dd802221e947e72a SystemUUID:ec29f4ae-e6f8-89f5-dd80-2221e947e72a BootID:546b548d-ec7e-445f-a673-0352fe144aa1 Filesystems:[{Device:/var/lib/kubelet/pods/f14812ec-6d3c-4957-be87-0454c8c21d65/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:465 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b9b99519ba00a1b4a23a36b03be1fdf331f77c56e50d2ac7f299ccc4f6ade62/userdata/shm DeviceMajor:0 DeviceMinor:478 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fee32755-d814-417f-8311-6a1dc5b60fa5/volumes/kubernetes.io~projected/kube-api-access-rj77k DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/kube-api-access-clwt9 DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~projected/kube-api-access-wf7dm DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c361feb9917e45dab6a40d71540b1bc98955f2846ed37b03787dad269bd5e4a4/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529/volumes/kubernetes.io~projected/kube-api-access-gmr5h DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/787eb665916078990674eb157d806f8a99ddbaf0b5ab0ac68d6556b7a25d91a6/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09934062-de9f-4e88-b25e-4f4a7947e4ce/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:475 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:503 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-511 DeviceMajor:0 DeviceMinor:511 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98bdb3de1e11569289361a6f2d17a2dd9c2753abafbf9349abbd09c33ff010a7/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21e4e1e7dab09e551ef7ef8b3a6d1c8b0a0de1a66aa69ce85ea84b2ec9237b8c/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5bf3124fa94c1795a0b3da7074abe2a8944352b96e58737d2ef1d13ab402056/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79c0a7af-0b6e-4056-8e2d-166885cb98dc/volumes/kubernetes.io~projected/kube-api-access-pkwx2 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb593b0969779537fd541df460e37740c48c8dbbf3f77bcb14c77cac2cf76046/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a83a01b9-9670-484c-ba66-658f61d5f162/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:434 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38807dac-4dc9-4d27-932b-e81189808eb1/volumes/kubernetes.io~projected/kube-api-access-lkjvm DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8b4d5b4eaa16a4737e886726db99da7327961d4b9d5dd0533cdc53f4e5f3f70/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9316842b-0133-456e-abeb-e0b38e7f04ff/volumes/kubernetes.io~projected/kube-api-access-hjtjc DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88456d88d676447b6fb0755c68a06c5e8b6d8ea2638401192324da10cb88ead2/userdata/shm DeviceMajor:0 DeviceMinor:476 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3785c7b8f0f4b2633c978829360bcc079fd9e5f5b02bd3c0aa39b015ae4add8/userdata/shm DeviceMajor:0 DeviceMinor:509 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volumes/kubernetes.io~projected/kube-api-access-jnwkw DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4428250-46e8-4f67-bb0c-285938ee2175/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-492 DeviceMajor:0 DeviceMinor:492 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77f7bca8-ea23-4b2b-9eaa-b26425121ff5/volumes/kubernetes.io~projected/kube-api-access-9f48r DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8057563-4fad-49a2-8d82-4902eca5fa57/volumes/kubernetes.io~projected/kube-api-access-g2h76 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0f42a0-fd35-4121-96b9-1d6363689977/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1/volumes/kubernetes.io~projected/kube-api-access-cwpcz DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cce826f4909ef00a95db549785d5d5f3a06415139943e2fa59de3d2f1eb5a25/userdata/shm DeviceMajor:0 DeviceMinor:459 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/455f84d7c40eb10d34fac51cc723697c75c82cc6aee4d565cd06aedb57d679bd/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f9fbd2fafd64df2cfe50c61ce867568b5fd152e6b73eaef39ff959c3bca5fc8/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ceda5fc6e9c6871ca5ca5788ad24b6b065aef2b7ac53bdc9f4763314b2cecf15/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/238ed5358e983b13e1f4e6c70c23cd89186d0980ed12c716297a0c2227b78555/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4090b54-5b6a-49f9-b092-d7c309f3d913/volumes/kubernetes.io~projected/kube-api-access-f88lv DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b634720f-79f7-4d34-a971-8d1ba231b3d3/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/139dd654d0ba8d107fd5dec6a7190a22918d33ef81002b33e4d8593f74306cad/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/942dd14b-9b63-4c8a-8362-218b9043307c/volumes/kubernetes.io~projected/kube-api-access-rvjq6 DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77f7bca8-ea23-4b2b-9eaa-b26425121ff5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:474 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae509a6d-ba84-42c0-9a51-0e7971cb9d68/volumes/kubernetes.io~projected/kube-api-access-csgw4 DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99c3425adc5ea75db0dbcb5b0472614400f8963bfb80490ee629088b304c5d52/userdata/shm DeviceMajor:0 DeviceMinor:470 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1fb504d1880f2e5fdc161ec56d44e5306c094421375ef07005ddd5b036f46ff/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2/volumes/kubernetes.io~projected/kube-api-access-mjcmw DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4428250-46e8-4f67-bb0c-285938ee2175/volumes/kubernetes.io~projected/kube-api-access-f8jxj DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/562095794d42c600a66f4491df56ef216cf5866a1b6c05240d0ea566189371d7/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a33cb5d1-e145-47db-9c98-2561a4213883/volumes/kubernetes.io~projected/kube-api-access-t8q8z DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8607ad7be1f639a72522fe3256f13461fc5e4c23e3f1a660e91809b30f61ce9c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38807dac-4dc9-4d27-932b-e81189808eb1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f14812ec-6d3c-4957-be87-0454c8c21d65/volumes/kubernetes.io~projected/kube-api-access-g5tkt DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0f42a0-fd35-4121-96b9-1d6363689977/volumes/kubernetes.io~projected/kube-api-access-lhqq5 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a4ff567ada319cd933ca254ed30e29c278c7c6bbb3367587b5905b3f8e22b23/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~projected/kube-api-access-75l4l DeviceMajor:0 DeviceMinor:508 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604f57d5-12dc-4ade-895d-c876688423fb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1974af933a7840ab6d3f926fbef7470df6de80d1394695647c1981e1b6429531/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c63af1f1-ad79-4694-955a-25eb509899d7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1e48c120d9af5509cb12c28d135e33e479156741fcbd392956ae5bab771233f/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48eb4acb-8de7-4b4e-9fea-ca7bf8556014/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:362 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/241ccb66-4fd3-427c-9b7f-279439013bfd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cca9f767847f4e5c430822b5e4c0ffd4d524069066f42804ba6b82651523f6e2/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/644ef22a9afc953fb3f4704023c5a386721cfee1fe596f82893e04f254f75168/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa58d08bd48ddd010c2eac4a2ec4af8f47902b4cc1186e99823eef9be713fd96/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b634720f-79f7-4d34-a971-8d1ba231b3d3/volumes/kubernetes.io~projected/kube-api-access-m5tzb DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/341fbe38-f97c-4a00-938b-6c4b9a30197e/volumes/kubernetes.io~projected/kube-api-access-kjb6d DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f67baedb4acce39a4212f5fdc7726fbcd15f0ab2cc8b6b5ff968137c8486a995/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be133361-4e22-4617-92fd-610a57bef9e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:507 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a345f30-ddf3-478a-81e0-5d50cbf38dc4/volumes/kubernetes.io~projected/kube-api-access-4fz89 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e93f8991e164e93bf181abdcaa43c1414300a76d57dc3bc293eb358881a8b188/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a345f30-ddf3-478a-81e0-5d50cbf38dc4/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/341fbe38-f97c-4a00-938b-6c4b9a30197e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/53a51104-b6b6-4c74-a3c0-e2c5c378f28f/volumes/kubernetes.io~projected/kube-api-access-9sz7n DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/584d682856321d23d7a19f19b0dc15aecd48054e6bc00e802f9eb621e3a8a7ae/userdata/shm DeviceMajor:0 DeviceMinor:438 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/af99842a-26b2-4e57-aae5-3aceef8f0776/volumes/kubernetes.io~projected/kube-api-access-54zw2 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes/kubernetes.io~projected/kube-api-access-4rwh6 DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f909c5ab4d06ac2df8ed042c8f12ac297041bf1b1f1c5b15be9924f92b22960b/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d172f407-b9ac-4193-b0f4-fab1c53e6874/volumes/kubernetes.io~projected/kube-api-access-tf86f DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8643bdf-eeca-4201-8d47-e4b1b69eefa2/volumes/kubernetes.io~projected/kube-api-access-8rpfb DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a83a01b9-9670-484c-ba66-658f61d5f162/volumes/kubernetes.io~projected/kube-api-access-9sg8v DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e/volumes/kubernetes.io~projected/kube-api-access-ffsxx DeviceMajor:0 DeviceMinor:455 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604f57d5-12dc-4ade-895d-c876688423fb/volumes/kubernetes.io~projected/kube-api-access-5pfnd DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d77d2c8b3f59aca5189494c2aae27361a48344f3aca5896afe11a935a00b979/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a4ff567ada319c MacAddress:ce:de:9f:67:55:6c Speed:10000 Mtu:8901} {Name:0d77d2c8b3f59ac MacAddress:56:ea:cb:a7:bb:e9 Speed:10000 Mtu:8901} {Name:139dd654d0ba8d1 MacAddress:9e:62:99:25:31:0b Speed:10000 Mtu:8901} {Name:1974af933a7840a MacAddress:a2:6d:99:fd:67:cd Speed:10000 Mtu:8901} {Name:21e4e1e7dab09e5 MacAddress:66:e7:69:21:c8:a0 Speed:10000 Mtu:8901} {Name:238ed5358e983b1 MacAddress:ee:c5:67:eb:06:c3 Speed:10000 Mtu:8901} {Name:3cce826f4909ef0 MacAddress:c6:4a:ed:e0:b3:ea Speed:10000 Mtu:8901} {Name:3e5cb5dc59ef42b MacAddress:3e:91:82:f9:45:a4 Speed:10000 Mtu:8901} {Name:4b9b99519ba00a1 MacAddress:f6:6c:cf:19:b0:9d Speed:10000 Mtu:8901} {Name:562095794d42c60 MacAddress:12:30:48:89:16:d6 Speed:10000 Mtu:8901} {Name:584d682856321d2 MacAddress:ca:6c:b1:53:52:10 Speed:10000 Mtu:8901} {Name:88456d88d676447 MacAddress:9e:b4:de:7f:3c:0f Speed:10000 Mtu:8901} {Name:98bdb3de1e11569 MacAddress:2a:05:e5:30:8b:20 Speed:10000 Mtu:8901} {Name:99c3425adc5ea75 MacAddress:66:ca:bc:eb:d2:69 Speed:10000 Mtu:8901} {Name:a1e48c120d9af55 MacAddress:52:6f:82:a9:cc:31 Speed:10000 Mtu:8901} {Name:b5bf3124fa94c17 MacAddress:8a:58:47:b9:28:e0 Speed:10000 Mtu:8901} {Name:bb593b096977953 MacAddress:ee:a4:b7:0e:d2:f8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ff:97:52:3d Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:70:0e:79:84:2b Speed:0 Mtu:8901} {Name:c361feb9917e45d MacAddress:56:1b:b9:94:eb:dc Speed:10000 Mtu:8901} {Name:cca9f767847f4e5 MacAddress:9e:f3:f1:c5:6f:43 Speed:10000 Mtu:8901} {Name:ceda5fc6e9c6871 MacAddress:ea:93:9d:44:ac:a0 Speed:10000 Mtu:8901} {Name:e8b4d5b4eaa16a4 MacAddress:7a:d9:61:92:96:3e Speed:10000 Mtu:8901} {Name:eb1954ed0a48f22 MacAddress:16:c6:c8:3c:ed:33 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ff:97:52:3d Speed:0 Mtu:9001} {Name:f67baedb4acce39 MacAddress:ca:dc:f0:63:30:21 Speed:10000 Mtu:8901} {Name:f909c5ab4d06ac2 MacAddress:36:3b:46:9e:6b:7a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6e:bf:48:6a:43:fb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:f4:b0:52:91:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:28:27.862776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.862779 5890 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:28:27.863076 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.862852 5890 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:28:27.863229 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863201 5890 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:28:27.863354 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863229 5890 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-105.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:28:27.863400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863356 5890 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:28:27.863400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863364 5890 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:28:27.863400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863375 5890 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:28:27.863400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863395 5890 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:28:27.863507 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863412 5890 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:28:27.863569 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863560 5890 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:28:27.863684 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863676 5890 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:28:27.863727 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863717 5890 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:28:27.863755 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863737 5890 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:28:27.863755 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863748 5890 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:28:27.863845 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863757 5890 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:28:27.863845 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.863764 5890 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:28:27.865107 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.865088 5890 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:28:27.865107 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.865106 5890 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:28:27.865638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.865613 5890 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:28:27.877928 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.877910 5890 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:28:27.877928 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.877927 5890 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:28:27.878058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878051 5890 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:28:27.878259 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878247 5890 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:28:27.878300 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878277 5890 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:28:27.878384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878370 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:28:27.878427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878392 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:28:27.878427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878401 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:28:27.878427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878407 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:28:27.878427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878413 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:28:27.878427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878420 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:28:27.878427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878426 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:28:27.878647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878431 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:28:27.878647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878438 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:28:27.878647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878444 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:28:27.878647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878462 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:28:27.878647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878473 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:28:27.878647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878505 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:28:27.878647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.878514 5890 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:28:27.882376 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.882358 5890 server.go:1263] "Started kubelet" Oct 31 00:28:27.886668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.882457 5890 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:28:27.886668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.882528 5890 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:28:27.886668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.883151 5890 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:28:27.883133 ip-10-0-143-105 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:28:27.887118 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.886814 5890 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:28:27.887118 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.887050 5890 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:28:27.889658 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.889639 5890 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:28:27.889819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.889683 5890 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:28:27.889899 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.889846 5890 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:28:27.889899 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.889861 5890 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:28:27.890044 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:27.889871 5890 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" Oct 31 00:28:27.890044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.889936 5890 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:28:27.890907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.890862 5890 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:22:17 +0000 UTC" deadline="2028-05-07 11:07:13.579659954 +0000 UTC" Oct 31 00:28:27.890995 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.890907 5890 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13306h38m45.688756726s" Oct 31 00:28:27.893188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893138 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4090b54-5b6a-49f9-b092-d7c309f3d913" volumeName="kubernetes.io/projected/a4090b54-5b6a-49f9-b092-d7c309f3d913-kube-api-access-f88lv" seLinuxMountContext="" Oct 31 00:28:27.893188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893178 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae509a6d-ba84-42c0-9a51-0e7971cb9d68" volumeName="kubernetes.io/projected/ae509a6d-ba84-42c0-9a51-0e7971cb9d68-kube-api-access-csgw4" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893200 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893214 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893225 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4428250-46e8-4f67-bb0c-285938ee2175" volumeName="kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893239 5890 factory.go:55] Registering systemd factory Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893255 5890 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893238 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341fbe38-f97c-4a00-938b-6c4b9a30197e" volumeName="kubernetes.io/secret/341fbe38-f97c-4a00-938b-6c4b9a30197e-serving-cert" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893301 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eb4acb-8de7-4b4e-9fea-ca7bf8556014" volumeName="kubernetes.io/configmap/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-nginx-conf" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893316 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529" volumeName="kubernetes.io/projected/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-kube-api-access-gmr5h" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893332 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529" volumeName="kubernetes.io/secret/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-key" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893345 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79c0a7af-0b6e-4056-8e2d-166885cb98dc" volumeName="kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893355 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79c0a7af-0b6e-4056-8e2d-166885cb98dc" volumeName="kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-binary-copy" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893364 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172f407-b9ac-4193-b0f4-fab1c53e6874" volumeName="kubernetes.io/configmap/d172f407-b9ac-4193-b0f4-fab1c53e6874-serviceca" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893373 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="942dd14b-9b63-4c8a-8362-218b9043307c" volumeName="kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6" seLinuxMountContext="" Oct 31 00:28:27.893373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893380 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341fbe38-f97c-4a00-938b-6c4b9a30197e" volumeName="kubernetes.io/projected/341fbe38-f97c-4a00-938b-6c4b9a30197e-kube-api-access-kjb6d" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893392 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604f57d5-12dc-4ade-895d-c876688423fb" volumeName="kubernetes.io/projected/604f57d5-12dc-4ade-895d-c876688423fb-kube-api-access-5pfnd" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893404 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604f57d5-12dc-4ade-895d-c876688423fb" volumeName="kubernetes.io/secret/604f57d5-12dc-4ade-895d-c876688423fb-serving-cert" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893425 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be133361-4e22-4617-92fd-610a57bef9e2" volumeName="kubernetes.io/empty-dir/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893438 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0f42a0-fd35-4121-96b9-1d6363689977" volumeName="kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-tmp" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893451 5890 factory.go:153] Registering CRI-O factory Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893464 5890 factory.go:223] Registration of the crio container factory successfully Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893506 5890 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893451 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893536 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893550 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be133361-4e22-4617-92fd-610a57bef9e2" volumeName="kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893559 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893567 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="241ccb66-4fd3-427c-9b7f-279439013bfd" volumeName="kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-tmp" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893575 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604f57d5-12dc-4ade-895d-c876688423fb" volumeName="kubernetes.io/configmap/604f57d5-12dc-4ade-895d-c876688423fb-config" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893588 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a345f30-ddf3-478a-81e0-5d50cbf38dc4" volumeName="kubernetes.io/projected/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-kube-api-access-4fz89" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893597 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8643bdf-eeca-4201-8d47-e4b1b69eefa2" volumeName="kubernetes.io/projected/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-kube-api-access-8rpfb" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893605 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b634720f-79f7-4d34-a971-8d1ba231b3d3" volumeName="kubernetes.io/configmap/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-config" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893625 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14812ec-6d3c-4957-be87-0454c8c21d65" volumeName="kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893640 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341fbe38-f97c-4a00-938b-6c4b9a30197e" volumeName="kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-trusted-ca" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893648 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53a51104-b6b6-4c74-a3c0-e2c5c378f28f" volumeName="kubernetes.io/projected/53a51104-b6b6-4c74-a3c0-e2c5c378f28f-kube-api-access-9sz7n" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893657 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0f42a0-fd35-4121-96b9-1d6363689977" volumeName="kubernetes.io/projected/7c0f42a0-fd35-4121-96b9-1d6363689977-kube-api-access-lhqq5" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893665 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="942dd14b-9b63-4c8a-8362-218b9043307c" volumeName="kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893682 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8643bdf-eeca-4201-8d47-e4b1b69eefa2" volumeName="kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893692 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b634720f-79f7-4d34-a971-8d1ba231b3d3" volumeName="kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls" seLinuxMountContext="" Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893524 5890 factory.go:103] Registering Raw factory Oct 31 00:28:27.893891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893736 5890 manager.go:1199] Started watching for new ooms in manager Oct 31 00:28:27.894590 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.894020 5890 manager.go:319] Starting recovery of all containers Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.893703 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79c0a7af-0b6e-4056-8e2d-166885cb98dc" volumeName="kubernetes.io/projected/79c0a7af-0b6e-4056-8e2d-166885cb98dc-kube-api-access-pkwx2" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895259 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0f42a0-fd35-4121-96b9-1d6363689977" volumeName="kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895299 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9316842b-0133-456e-abeb-e0b38e7f04ff" volumeName="kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-daemon-config" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895314 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8057563-4fad-49a2-8d82-4902eca5fa57" volumeName="kubernetes.io/projected/c8057563-4fad-49a2-8d82-4902eca5fa57-kube-api-access-g2h76" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895331 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4428250-46e8-4f67-bb0c-285938ee2175" volumeName="kubernetes.io/projected/f4428250-46e8-4f67-bb0c-285938ee2175-kube-api-access-f8jxj" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895354 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09934062-de9f-4e88-b25e-4f4a7947e4ce" volumeName="kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895368 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" volumeName="kubernetes.io/projected/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-kube-api-access-9f48r" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895387 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af99842a-26b2-4e57-aae5-3aceef8f0776" volumeName="kubernetes.io/projected/af99842a-26b2-4e57-aae5-3aceef8f0776-kube-api-access-54zw2" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895402 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895415 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172f407-b9ac-4193-b0f4-fab1c53e6874" volumeName="kubernetes.io/projected/d172f407-b9ac-4193-b0f4-fab1c53e6874-kube-api-access-tf86f" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895433 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14812ec-6d3c-4957-be87-0454c8c21d65" volumeName="kubernetes.io/empty-dir/f14812ec-6d3c-4957-be87-0454c8c21d65-tmp-dir" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895447 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eb4acb-8de7-4b4e-9fea-ca7bf8556014" volumeName="kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895478 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799b2dd6-6d05-4ca1-bc09-79bd1634fbaa" volumeName="kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-env-overrides" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895504 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0f42a0-fd35-4121-96b9-1d6363689977" volumeName="kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-service-ca-bundle" seLinuxMountContext="" Oct 31 00:28:27.895678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895521 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8188efe6-6d2b-46cc-ba95-7782ea3a4e6e" volumeName="kubernetes.io/projected/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e-kube-api-access-ffsxx" seLinuxMountContext="" Oct 31 00:28:27.898606 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.895540 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.898620 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83a01b9-9670-484c-ba66-658f61d5f162" volumeName="kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.898888 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b634720f-79f7-4d34-a971-8d1ba231b3d3" volumeName="kubernetes.io/projected/b634720f-79f7-4d34-a971-8d1ba231b3d3-kube-api-access-m5tzb" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.898909 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.898932 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799b2dd6-6d05-4ca1-bc09-79bd1634fbaa" volumeName="kubernetes.io/secret/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.898957 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529" volumeName="kubernetes.io/configmap/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-cabundle" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.898975 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0f42a0-fd35-4121-96b9-1d6363689977" volumeName="kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-snapshots" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899012 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899026 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899044 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="942dd14b-9b63-4c8a-8362-218b9043307c" volumeName="kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899058 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="942dd14b-9b63-4c8a-8362-218b9043307c" volumeName="kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899078 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83a01b9-9670-484c-ba66-658f61d5f162" volumeName="kubernetes.io/projected/a83a01b9-9670-484c-ba66-658f61d5f162-kube-api-access-9sg8v" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899096 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a33cb5d1-e145-47db-9c98-2561a4213883" volumeName="kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899110 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8643bdf-eeca-4201-8d47-e4b1b69eefa2" volumeName="kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-default-certificate" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899129 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8643bdf-eeca-4201-8d47-e4b1b69eefa2" volumeName="kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899158 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63af1f1-ad79-4694-955a-25eb509899d7" volumeName="kubernetes.io/configmap/c63af1f1-ad79-4694-955a-25eb509899d7-konnectivity-ca" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899173 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63af1f1-ad79-4694-955a-25eb509899d7" volumeName="kubernetes.io/secret/c63af1f1-ad79-4694-955a-25eb509899d7-agent-certs" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899191 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14812ec-6d3c-4957-be87-0454c8c21d65" volumeName="kubernetes.io/configmap/f14812ec-6d3c-4957-be87-0454c8c21d65-config-volume" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899206 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="241ccb66-4fd3-427c-9b7f-279439013bfd" volumeName="kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-tuned" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899240 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38807dac-4dc9-4d27-932b-e81189808eb1" volumeName="kubernetes.io/configmap/38807dac-4dc9-4d27-932b-e81189808eb1-config" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899254 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38807dac-4dc9-4d27-932b-e81189808eb1" volumeName="kubernetes.io/projected/38807dac-4dc9-4d27-932b-e81189808eb1-kube-api-access-lkjvm" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899273 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799b2dd6-6d05-4ca1-bc09-79bd1634fbaa" volumeName="kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-config" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899286 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799b2dd6-6d05-4ca1-bc09-79bd1634fbaa" volumeName="kubernetes.io/projected/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-kube-api-access-jnwkw" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899299 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899318 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83a01b9-9670-484c-ba66-658f61d5f162" volumeName="kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899334 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af99842a-26b2-4e57-aae5-3aceef8f0776" volumeName="kubernetes.io/empty-dir/af99842a-26b2-4e57-aae5-3aceef8f0776-tmp-dir" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899351 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899365 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fcf1ba8-6176-418b-8477-6f5b89a16a41" volumeName="kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899378 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be133361-4e22-4617-92fd-610a57bef9e2" volumeName="kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-metrics-client-ca" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899398 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8057563-4fad-49a2-8d82-4902eca5fa57" volumeName="kubernetes.io/configmap/c8057563-4fad-49a2-8d82-4902eca5fa57-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899412 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899431 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4428250-46e8-4f67-bb0c-285938ee2175" volumeName="kubernetes.io/configmap/f4428250-46e8-4f67-bb0c-285938ee2175-telemetry-config" seLinuxMountContext="" Oct 31 00:28:27.899481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.899447 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fee32755-d814-417f-8311-6a1dc5b60fa5" volumeName="kubernetes.io/projected/fee32755-d814-417f-8311-6a1dc5b60fa5-kube-api-access-rj77k" seLinuxMountContext="" Oct 31 00:28:27.900932 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.900905 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799b2dd6-6d05-4ca1-bc09-79bd1634fbaa" volumeName="kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:28:27.900932 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.900932 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e5eb4b-7a3b-43d5-81f9-cda790ace0b2" volumeName="kubernetes.io/projected/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2-kube-api-access-mjcmw" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.900955 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a345f30-ddf3-478a-81e0-5d50cbf38dc4" volumeName="kubernetes.io/secret/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-cert" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.900968 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0f42a0-fd35-4121-96b9-1d6363689977" volumeName="kubernetes.io/secret/7c0f42a0-fd35-4121-96b9-1d6363689977-serving-cert" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.900981 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83a01b9-9670-484c-ba66-658f61d5f162" volumeName="kubernetes.io/empty-dir/a83a01b9-9670-484c-ba66-658f61d5f162-data-volume" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.900993 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901004 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" volumeName="kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901016 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="241ccb66-4fd3-427c-9b7f-279439013bfd" volumeName="kubernetes.io/projected/241ccb66-4fd3-427c-9b7f-279439013bfd-kube-api-access-wf7dm" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901058 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341fbe38-f97c-4a00-938b-6c4b9a30197e" volumeName="kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-config" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901072 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9316842b-0133-456e-abeb-e0b38e7f04ff" volumeName="kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-cni-binary-copy" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901083 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9316842b-0133-456e-abeb-e0b38e7f04ff" volumeName="kubernetes.io/projected/9316842b-0133-456e-abeb-e0b38e7f04ff-kube-api-access-hjtjc" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901094 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be133361-4e22-4617-92fd-610a57bef9e2" volumeName="kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-tls" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901108 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14812ec-6d3c-4957-be87-0454c8c21d65" volumeName="kubernetes.io/projected/f14812ec-6d3c-4957-be87-0454c8c21d65-kube-api-access-g5tkt" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901119 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77f7bca8-ea23-4b2b-9eaa-b26425121ff5" volumeName="kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901131 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8643bdf-eeca-4201-8d47-e4b1b69eefa2" volumeName="kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-stats-auth" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901142 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be133361-4e22-4617-92fd-610a57bef9e2" volumeName="kubernetes.io/projected/be133361-4e22-4617-92fd-610a57bef9e2-kube-api-access-75l4l" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901154 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be133361-4e22-4617-92fd-610a57bef9e2" volumeName="kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901168 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1caab5f-35c5-4f4d-8f81-0d18527a7ef1" volumeName="kubernetes.io/projected/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1-kube-api-access-cwpcz" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901180 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38807dac-4dc9-4d27-932b-e81189808eb1" volumeName="kubernetes.io/secret/38807dac-4dc9-4d27-932b-e81189808eb1-serving-cert" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901191 5890 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79c0a7af-0b6e-4056-8e2d-166885cb98dc" volumeName="kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901203 5890 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:28:27.901459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.901210 5890 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:28:27.908059 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.908035 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:28:27.910095 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.910075 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:28:27.915217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.915189 5890 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:28:27.924410 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.924385 5890 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:28:27.924521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.924511 5890 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:28:27.924620 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.924611 5890 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:28:27.924739 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:27.924721 5890 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:28:27.951769 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.951737 5890 generic.go:356] "Generic (PLEG): container finished" podID="63412fdebab41a737ab007fe557fdbf2" containerID="ea1ab6c01ff87eaad01c49568b716dc5cf28ad59da6a7b820856a48ce5730b6f" exitCode=0 Oct 31 00:28:27.964680 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.964649 5890 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="2a019323adb1e7eb0147412bc95d3f12c26e90b89fa840735f8f3df310ec6d48" exitCode=0 Oct 31 00:28:27.964680 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.964678 5890 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="34e61dfdbc07a403031459146690a2a8fbecbc259ca203b97bcc32e19ebb5142" exitCode=0 Oct 31 00:28:27.964869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.964691 5890 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="5099a3da421312b4f4f21ed80fabe512064ecb586f76d924fffcb8131692de06" exitCode=0 Oct 31 00:28:27.964869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.964701 5890 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="ec9ca3302c2f66387722b85ad8d44379dcb6b0e91be9418fa842404ef8a766f6" exitCode=0 Oct 31 00:28:27.964869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.964712 5890 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="2d77070633117af176681e84584626d81a6757b8786b72804ec175b4836d3863" exitCode=0 Oct 31 00:28:27.964869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.964720 5890 generic.go:356] "Generic (PLEG): container finished" podID="79c0a7af-0b6e-4056-8e2d-166885cb98dc" containerID="38546f83033367389251155f2c498cfd2932f4e02da4817865303265a65346b3" exitCode=0 Oct 31 00:28:27.976144 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.976086 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:28:27.976144 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:27.976117 5890 generic.go:356] "Generic (PLEG): container finished" podID="341fbe38-f97c-4a00-938b-6c4b9a30197e" containerID="befbbec3de1d94bdec332280738b16998f7aec9d2a0cbd2a430ceb2330c90e1b" exitCode=255 Oct 31 00:28:27.990004 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:27.989969 5890 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" Oct 31 00:28:28.018263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.018238 5890 manager.go:324] Recovery completed Oct 31 00:28:28.024960 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:28.024935 5890 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 00:28:28.042515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.042488 5890 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:28.044285 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.044265 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:28.044385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.044296 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:28.044385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.044307 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:28.047325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047270 5890 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:28:28.047325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047287 5890 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:28:28.047325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047301 5890 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:28.047552 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047308 5890 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:28:28.047617 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047565 5890 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:28:28.047617 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047583 5890 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:28:28.047617 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047607 5890 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:28:28.047617 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047616 5890 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:28:28.047757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047626 5890 policy_none.go:50] "Start" Oct 31 00:28:28.047757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047640 5890 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:28:28.047757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047656 5890 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:28:28.047885 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047870 5890 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:28:28.047885 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047880 5890 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:28:28.047885 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.047886 5890 policy_none.go:44] "Start" Oct 31 00:28:28.048284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.048267 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:28.048346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.048294 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:28.048346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.048306 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:28.053287 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.053261 5890 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:28:28.055911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.055891 5890 manager.go:341] "Starting Device Plugin manager" Oct 31 00:28:28.055992 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:28.055922 5890 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:28:28.055992 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.055932 5890 server.go:87] "Starting device plugin registration server" Oct 31 00:28:28.056259 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.056244 5890 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:28:28.056318 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.056260 5890 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:28:28.056404 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.056383 5890 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:28:28.056549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.056507 5890 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:28:28.056672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.056554 5890 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:28:28.056672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.056562 5890 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:28:28.056779 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.056609 5890 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:28:28.056543242Z","UUID":"d8f00283-d0f9-4233-a542-eee066eb42ec","Handler":null,"Name":"","Endpoint":""} Oct 31 00:28:28.058430 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.058410 5890 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:28:28.058522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.058439 5890 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:28:28.058604 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:28.058585 5890 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:28:28.058682 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:28.058629 5890 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-105.us-west-2.compute.internal\" not found" Oct 31 00:28:28.079822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.071035 5890 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:28.079822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.071430 5890 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:28.079822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.071821 5890 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:28.079822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.072174 5890 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:28:28.157025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.156993 5890 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:28.158086 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.158059 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:28.158182 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.158095 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:28.158182 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.158120 5890 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-105.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:28.158182 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.158167 5890 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.165521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.165501 5890 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:28.225269 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.225220 5890 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal"] Oct 31 00:28:28.225907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.225871 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"63412fdebab41a737ab007fe557fdbf2","Type":"ContainerStarted","Data":"d942cbe568e12cdb19c5a7155eaad526b995a06c4f9d4ba01420bb6863d78bad"} Oct 31 00:28:28.226028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.225916 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"63412fdebab41a737ab007fe557fdbf2","Type":"ContainerDied","Data":"ea1ab6c01ff87eaad01c49568b716dc5cf28ad59da6a7b820856a48ce5730b6f"} Oct 31 00:28:28.226028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.225939 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"63412fdebab41a737ab007fe557fdbf2","Type":"ContainerStarted","Data":"5e19dbe1c50991c426ce5d9f66ee6b40377feccbe2dde31ad2f2cdffb719de5f"} Oct 31 00:28:28.226028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.225998 5890 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.226202 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.226078 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"d22eb601a1d03eddfb99de6ecc9e9dbd","Type":"ContainerStarted","Data":"bdc95e6f9a1357e180acd46be9a61cf74ca6d28325b7c40b47507eb99ae091df"} Oct 31 00:28:28.226202 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.226087 5890 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.226202 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.226099 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" event={"ID":"d22eb601a1d03eddfb99de6ecc9e9dbd","Type":"ContainerStarted","Data":"34518891b1b118b8265d611daf1f41da7b44f7e0750f39c47667ca3438691f4b"} Oct 31 00:28:28.263354 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.263332 5890 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.263441 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.263399 5890 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.288493 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.288465 5890 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:28.288590 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:28.288515 5890 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.290560 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.290546 5890 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:28.290646 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:28.290586 5890 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.373154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.373120 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.373311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.373173 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.373311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.373191 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d22eb601a1d03eddfb99de6ecc9e9dbd-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"d22eb601a1d03eddfb99de6ecc9e9dbd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.474322 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.474288 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.474322 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.474245 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.474549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.474353 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.474549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.474386 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d22eb601a1d03eddfb99de6ecc9e9dbd-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"d22eb601a1d03eddfb99de6ecc9e9dbd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.474549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.474423 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d22eb601a1d03eddfb99de6ecc9e9dbd-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"d22eb601a1d03eddfb99de6ecc9e9dbd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.474549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.474451 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63412fdebab41a737ab007fe557fdbf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" (UID: \"63412fdebab41a737ab007fe557fdbf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:28.864905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.864867 5890 apiserver.go:52] "Watching apiserver" Oct 31 00:28:28.966124 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.966087 5890 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:28:28.966848 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.966813 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z4v42","openshift-ingress/router-default-5cfd77cbdf-kcvgr","openshift-multus/multus-additional-cni-plugins-zz7n4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-mf88n","openshift-network-console/networking-console-plugin-86c478bb88-4w8gv","kube-system/global-pull-secret-syncer-ksxrk","openshift-cluster-node-tuning-operator/tuned-mdpsp","openshift-console-operator/console-operator-7cd56b8dbc-jlcr7","openshift-ingress-canary/ingress-canary-wqdmn","openshift-network-diagnostics/network-check-target-8grlc","openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k","kube-system/kas-connection-checker-685895c7c9-prfjr","openshift-insights/insights-runtime-extractor-jd4rk","kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal","openshift-insights/insights-operator-7c4986d449-sbz69","openshift-monitoring/node-exporter-9wfdm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk","openshift-image-registry/node-ca-fgvdh","kube-system/kas-connection-checker-685895c7c9-n7zhr","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls","openshift-image-registry/image-registry-7c65457c94-5cwm4","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6","openshift-network-operator/iptables-alerter-qlblw","openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72","openshift-ovn-kubernetes/ovnkube-node-c2rj2","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl","openshift-dns/node-resolver-946fm","openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x","kube-system/konnectivity-agent-nt477","kube-system/kas-connection-checker-685895c7c9-24h4x","openshift-multus/multus-nn2wc","openshift-image-registry/image-registry-5b88d6f687-f96vv","openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d","openshift-service-ca/service-ca-5fffc97bdf-427g5","openshift-insights/periodic-gathering-bcqhp-55jrh"] Oct 31 00:28:28.991772 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:28.991746 5890 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:28:29.006663 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.006634 5890 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:29.006802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.006742 5890 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:29.029274 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.029247 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.029406 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.029391 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.029481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.029451 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 00:28:29.029573 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.029563 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.029672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.029655 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.029930 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.029891 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.030152 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.030122 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.030258 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.030235 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.030609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.030595 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 00:28:29.030916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.030899 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:28:29.031130 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.031115 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mpt2z\"" Oct 31 00:28:29.031178 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.031165 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 00:28:29.047340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047316 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nl57f\"" Oct 31 00:28:29.047476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047316 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:28:29.047476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047260 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 00:28:29.047476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047362 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-c9dtf\"" Oct 31 00:28:29.047476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047430 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.047476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047404 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.047702 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047488 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 00:28:29.047702 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.047573 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rbl9c\"" Oct 31 00:28:29.077525 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077494 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.077698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077533 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-root\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.077698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077562 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysconfig\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.077698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077591 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-config\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.077698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077644 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.077698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077683 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-cni-binary-copy\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.077943 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077833 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-etc-kubernetes\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.077943 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077863 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.077943 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077908 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-config\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.078082 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.077980 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.078082 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078046 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 00:28:29.078082 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078072 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovn-node-metrics-cert\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.078268 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078107 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.078268 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078131 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-run\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.078268 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078146 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 00:28:29.078268 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078173 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88lv\" (UniqueName: \"kubernetes.io/projected/a4090b54-5b6a-49f9-b092-d7c309f3d913-kube-api-access-f88lv\") pod \"kas-connection-checker-685895c7c9-n7zhr\" (UID: \"a4090b54-5b6a-49f9-b092-d7c309f3d913\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" Oct 31 00:28:29.078437 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078288 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:29.078437 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078335 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.078437 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078377 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.078437 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078409 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpfb\" (UniqueName: \"kubernetes.io/projected/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-kube-api-access-8rpfb\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078452 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-systemd\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078481 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8057563-4fad-49a2-8d82-4902eca5fa57-host-slash\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078509 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078539 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078561 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078571 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4428250-46e8-4f67-bb0c-285938ee2175-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078598 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.078630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078621 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078645 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-snapshots\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078690 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-netns\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078723 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-system-cni-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078730 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-snapshots\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078749 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-tls\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078821 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf86f\" (UniqueName: \"kubernetes.io/projected/d172f407-b9ac-4193-b0f4-fab1c53e6874-kube-api-access-tf86f\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078834 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4428250-46e8-4f67-bb0c-285938ee2175-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:29.078968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078917 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-multus\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.079240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078992 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-sys\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.079240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.078995 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.079240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079030 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgw4\" (UniqueName: \"kubernetes.io/projected/ae509a6d-ba84-42c0-9a51-0e7971cb9d68-kube-api-access-csgw4\") pod \"kas-connection-checker-685895c7c9-24h4x\" (UID: \"ae509a6d-ba84-42c0-9a51-0e7971cb9d68\") " pod="kube-system/kas-connection-checker-685895c7c9-24h4x" Oct 31 00:28:29.079240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079068 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.079240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079094 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:28:29.079240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079172 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pfnd\" (UniqueName: \"kubernetes.io/projected/604f57d5-12dc-4ade-895d-c876688423fb-kube-api-access-5pfnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:28:29.079240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079211 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-multus-certs\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.079526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079244 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.079526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079294 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.079526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079326 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-wtmp\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.079526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079353 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-stats-auth\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.079526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079405 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.079526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079441 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:28:29.079526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079471 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.079811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079602 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f48r\" (UniqueName: \"kubernetes.io/projected/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-kube-api-access-9f48r\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:28:29.079811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079629 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d172f407-b9ac-4193-b0f4-fab1c53e6874-host\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:28:29.079811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079646 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-tmp\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.079811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079667 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-daemon-config\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.079811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079688 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2h76\" (UniqueName: \"kubernetes.io/projected/c8057563-4fad-49a2-8d82-4902eca5fa57-kube-api-access-g2h76\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:28:29.079811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079722 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.079811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079779 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079823 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83a01b9-9670-484c-ba66-658f61d5f162-data-volume\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079849 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-tuned\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079870 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c0f42a0-fd35-4121-96b9-1d6363689977-tmp\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079880 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38807dac-4dc9-4d27-932b-e81189808eb1-serving-cert\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079905 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zw2\" (UniqueName: \"kubernetes.io/projected/af99842a-26b2-4e57-aae5-3aceef8f0776-kube-api-access-54zw2\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079939 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83a01b9-9670-484c-ba66-658f61d5f162-data-volume\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079940 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-registration-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079952 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-tuned\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.079982 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-host\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080070 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080094 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-hostroot\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080131 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.080179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080164 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-cnibin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080231 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080280 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcmw\" (UniqueName: \"kubernetes.io/projected/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2-kube-api-access-mjcmw\") pod \"kas-connection-checker-685895c7c9-prfjr\" (UID: \"07e5eb4b-7a3b-43d5-81f9-cda790ace0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-prfjr" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080319 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-kubelet-config\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080394 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-log-socket\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080442 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-modprobe-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080468 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clwt9\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080531 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080574 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sg8v\" (UniqueName: \"kubernetes.io/projected/a83a01b9-9670-484c-ba66-658f61d5f162-kube-api-access-9sg8v\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080632 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-var-lib-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080653 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-node-log\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080671 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af99842a-26b2-4e57-aae5-3aceef8f0776-hosts-file\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:28:29.080703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080694 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-script-lib\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080728 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99842a-26b2-4e57-aae5-3aceef8f0776-tmp-dir\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080759 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-etc-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080815 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63af1f1-ad79-4694-955a-25eb509899d7-konnectivity-ca\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080871 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99842a-26b2-4e57-aae5-3aceef8f0776-tmp-dir\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080893 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080911 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cnibin\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080928 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-env-overrides\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.080979 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-var-lib-kubelet\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081013 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081037 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081065 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-k8s-cni-cncf-io\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081111 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtjc\" (UniqueName: \"kubernetes.io/projected/9316842b-0133-456e-abeb-e0b38e7f04ff-kube-api-access-hjtjc\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.081131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081133 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081150 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081170 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081265 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081289 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-conf-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081307 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-metrics-client-ca\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081340 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83a01b9-9670-484c-ba66-658f61d5f162-crio-socket\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081366 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-device-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081386 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081411 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqq5\" (UniqueName: \"kubernetes.io/projected/7c0f42a0-fd35-4121-96b9-1d6363689977-kube-api-access-lhqq5\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081460 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjb6d\" (UniqueName: \"kubernetes.io/projected/341fbe38-f97c-4a00-938b-6c4b9a30197e-kube-api-access-kjb6d\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081503 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081520 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081537 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-system-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081553 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081581 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-slash\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081605 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-netd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.081668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081634 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081692 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081729 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj77k\" (UniqueName: \"kubernetes.io/projected/fee32755-d814-417f-8311-6a1dc5b60fa5-kube-api-access-rj77k\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081760 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sz7n\" (UniqueName: \"kubernetes.io/projected/53a51104-b6b6-4c74-a3c0-e2c5c378f28f-kube-api-access-9sz7n\") pod \"volume-data-source-validator-58c9f5f46d-86qq6\" (UID: \"53a51104-b6b6-4c74-a3c0-e2c5c378f28f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081857 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-key\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081894 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tkt\" (UniqueName: \"kubernetes.io/projected/f14812ec-6d3c-4957-be87-0454c8c21d65-kube-api-access-g5tkt\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081993 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b634720f-79f7-4d34-a971-8d1ba231b3d3-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.081999 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsxx\" (UniqueName: \"kubernetes.io/projected/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e-kube-api-access-ffsxx\") pod \"migrator-66df5d496b-89n4d\" (UID: \"8188efe6-6d2b-46cc-ba95-7782ea3a4e6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082039 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-config\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082061 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8057563-4fad-49a2-8d82-4902eca5fa57-iptables-alerter-script\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082084 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c0f42a0-fd35-4121-96b9-1d6363689977-serving-cert\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082128 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fz89\" (UniqueName: \"kubernetes.io/projected/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-kube-api-access-4fz89\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:28:29.082311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082299 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-tmp\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082339 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwh6\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082371 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/604f57d5-12dc-4ade-895d-c876688423fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082404 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241ccb66-4fd3-427c-9b7f-279439013bfd-tmp\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082404 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38807dac-4dc9-4d27-932b-e81189808eb1-config\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082452 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d172f407-b9ac-4193-b0f4-fab1c53e6874-serviceca\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082473 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-socket-dir-parent\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082488 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-ovn\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082508 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-sys\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082546 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604f57d5-12dc-4ade-895d-c876688423fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082551 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c0f42a0-fd35-4121-96b9-1d6363689977-serving-cert\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082566 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jxj\" (UniqueName: \"kubernetes.io/projected/f4428250-46e8-4f67-bb0c-285938ee2175-kube-api-access-f8jxj\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:29.082625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082607 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-kubelet\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082640 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-dbus\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082671 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082712 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-bin\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082761 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-socket-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082794 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-kubernetes\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082850 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63af1f1-ad79-4694-955a-25eb509899d7-agent-certs\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082873 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-cabundle\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082890 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082913 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-lib-modules\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082944 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.082978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082963 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.082984 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-conf\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083012 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjq6\" (UniqueName: \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083028 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-bin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083032 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083045 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-textfile\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083092 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75l4l\" (UniqueName: \"kubernetes.io/projected/be133361-4e22-4617-92fd-610a57bef9e2-kube-api-access-75l4l\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083118 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083145 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083152 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-textfile\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083174 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjvm\" (UniqueName: \"kubernetes.io/projected/38807dac-4dc9-4d27-932b-e81189808eb1-kube-api-access-lkjvm\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083205 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083221 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-default-certificate\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083225 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083243 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-cert\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083267 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/341fbe38-f97c-4a00-938b-6c4b9a30197e-serving-cert\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.083341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083292 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7dm\" (UniqueName: \"kubernetes.io/projected/241ccb66-4fd3-427c-9b7f-279439013bfd-kube-api-access-wf7dm\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083396 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpcz\" (UniqueName: \"kubernetes.io/projected/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1-kube-api-access-cwpcz\") pod \"network-check-source-6f78d59f99-cz22k\" (UID: \"c1caab5f-35c5-4f4d-8f81-0d18527a7ef1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083444 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-kube-api-access-gmr5h\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083473 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwx2\" (UniqueName: \"kubernetes.io/projected/79c0a7af-0b6e-4056-8e2d-166885cb98dc-kube-api-access-pkwx2\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083495 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083500 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f14812ec-6d3c-4957-be87-0454c8c21d65-config-volume\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083520 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/341fbe38-f97c-4a00-938b-6c4b9a30197e-serving-cert\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083526 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-kubelet\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083547 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-netns\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083581 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-systemd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083631 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083657 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14812ec-6d3c-4957-be87-0454c8c21d65-tmp-dir\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083675 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tzb\" (UniqueName: \"kubernetes.io/projected/b634720f-79f7-4d34-a971-8d1ba231b3d3-kube-api-access-m5tzb\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083694 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-systemd-units\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083711 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-sys-fs\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083744 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14812ec-6d3c-4957-be87-0454c8c21d65-tmp-dir\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083808 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083827 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-service-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083845 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwkw\" (UniqueName: \"kubernetes.io/projected/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-kube-api-access-jnwkw\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083862 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083879 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4428250-46e8-4f67-bb0c-285938ee2175-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:29.084044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.083953 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.084618 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.084253 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c0f42a0-fd35-4121-96b9-1d6363689977-service-ca-bundle\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.088555 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.088537 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/341fbe38-f97c-4a00-938b-6c4b9a30197e-trusted-ca\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.092417 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.092402 5890 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:29.092465 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:29.092446 5890 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:29.092503 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.092402 5890 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:29.092537 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:29.092516 5890 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal" Oct 31 00:28:29.141538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141403 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 00:28:29.141538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141442 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 00:28:29.141538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141454 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.141538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141498 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141573 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141621 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141654 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141691 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141703 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141708 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141728 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.141810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141737 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141822 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141858 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141929 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141935 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141944 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141961 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.141986 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142043 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142109 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:28:29.142117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142118 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142173 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142185 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142190 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142235 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142259 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142275 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142309 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142351 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142389 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142392 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142407 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142436 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142446 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142447 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.142537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142539 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142590 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142602 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142629 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38807dac-4dc9-4d27-932b-e81189808eb1-config\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142634 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142673 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142697 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142723 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142740 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142750 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142770 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cdw8b\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142753 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-78fsr\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142772 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142826 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604f57d5-12dc-4ade-895d-c876688423fb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142828 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jsqnl\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142871 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2znq\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142881 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142887 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142917 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nwzth\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.142991 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143017 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dwmlr\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143044 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143114 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m2wdr\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143124 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143127 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-cabundle\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143151 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143163 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-trjc8\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143170 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63af1f1-ad79-4694-955a-25eb509899d7-agent-certs\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143291 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:28:29.143291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143294 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d172f407-b9ac-4193-b0f4-fab1c53e6874-serviceca\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143381 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zhjnq\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143415 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143443 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143444 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-default-certificate\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143668 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hg2s2\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143696 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143779 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f14812ec-6d3c-4957-be87-0454c8c21d65-config-volume\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143869 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.143937 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sqldj\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144013 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144025 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-cert\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144070 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144117 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jw542\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144146 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5xshs\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144161 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144196 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144209 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52f7q\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144244 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144259 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf86f\" (UniqueName: \"kubernetes.io/projected/d172f407-b9ac-4193-b0f4-fab1c53e6874-kube-api-access-tf86f\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144281 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144298 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88lv\" (UniqueName: \"kubernetes.io/projected/a4090b54-5b6a-49f9-b092-d7c309f3d913-kube-api-access-f88lv\") pod \"kas-connection-checker-685895c7c9-n7zhr\" (UID: \"a4090b54-5b6a-49f9-b092-d7c309f3d913\") " pod="kube-system/kas-connection-checker-685895c7c9-n7zhr" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144318 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144358 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144360 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144409 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9qgw5\"" Oct 31 00:28:29.144447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144415 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pfnd\" (UniqueName: \"kubernetes.io/projected/604f57d5-12dc-4ade-895d-c876688423fb-kube-api-access-5pfnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144479 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ghdng\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144715 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgw4\" (UniqueName: \"kubernetes.io/projected/ae509a6d-ba84-42c0-9a51-0e7971cb9d68-kube-api-access-csgw4\") pod \"kas-connection-checker-685895c7c9-24h4x\" (UID: \"ae509a6d-ba84-42c0-9a51-0e7971cb9d68\") " pod="kube-system/kas-connection-checker-685895c7c9-24h4x" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144763 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8q8z\" (UniqueName: \"kubernetes.io/projected/a33cb5d1-e145-47db-9c98-2561a4213883-kube-api-access-t8q8z\") pod \"network-check-target-8grlc\" (UID: \"a33cb5d1-e145-47db-9c98-2561a4213883\") " pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.144954 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7hhvp\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145040 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-85rmc\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145081 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rw8bx\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145120 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145144 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-65vz6\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145153 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rlgrr\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145169 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6gdh\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145189 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145193 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145210 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145249 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lwt4s\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145258 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145524 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpfb\" (UniqueName: \"kubernetes.io/projected/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-kube-api-access-8rpfb\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.145540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145544 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r5jc7\"" Oct 31 00:28:29.146005 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145648 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:28:29.146005 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.145910 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-46b97\"" Oct 31 00:28:29.148724 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.148704 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.148724 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.148719 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.148893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.148728 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovn-node-metrics-cert\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.148893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.148706 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.148893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.148762 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-cni-binary-copy\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.148893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.148764 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.148893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.148873 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.149946 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.149960 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-stats-auth\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.149964 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.149994 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-daemon-config\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150001 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-metrics-certs\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150012 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-metrics-certs\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150022 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-tls\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.150058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150033 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83a01b9-9670-484c-ba66-658f61d5f162-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.150442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150093 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.150442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150170 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f14812ec-6d3c-4957-be87-0454c8c21d65-metrics-tls\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.150442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150171 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:28:29.150442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150315 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38807dac-4dc9-4d27-932b-e81189808eb1-serving-cert\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:28:29.150442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150333 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.150442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.150349 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.151344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.151320 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-script-lib\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.151420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.151330 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eb4acb-8de7-4b4e-9fea-ca7bf8556014-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4w8gv\" (UID: \"48eb4acb-8de7-4b4e-9fea-ca7bf8556014\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4w8gv" Oct 31 00:28:29.151458 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.151415 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.151458 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.151420 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63af1f1-ad79-4694-955a-25eb509899d7-konnectivity-ca\") pod \"konnectivity-agent-nt477\" (UID: \"c63af1f1-ad79-4694-955a-25eb509899d7\") " pod="kube-system/konnectivity-agent-nt477" Oct 31 00:28:29.151596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.151503 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83a01b9-9670-484c-ba66-658f61d5f162-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.152020 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.151993 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09934062-de9f-4e88-b25e-4f4a7947e4ce-original-pull-secret\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.152117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.152058 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be133361-4e22-4617-92fd-610a57bef9e2-metrics-client-ca\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.152173 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.152129 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-env-overrides\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.152238 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-signing-key\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.152288 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-ovnkube-config\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.152348 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.152364 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.152428 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8057563-4fad-49a2-8d82-4902eca5fa57-iptables-alerter-script\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.153146 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.153457 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/604f57d5-12dc-4ade-895d-c876688423fb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-97sls\" (UID: \"604f57d5-12dc-4ade-895d-c876688423fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.153634 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f48r\" (UniqueName: \"kubernetes.io/projected/77f7bca8-ea23-4b2b-9eaa-b26425121ff5-kube-api-access-9f48r\") pod \"network-metrics-daemon-mf88n\" (UID: \"77f7bca8-ea23-4b2b-9eaa-b26425121ff5\") " pod="openshift-multus/network-metrics-daemon-mf88n" Oct 31 00:28:29.155997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.153665 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8643bdf-eeca-4201-8d47-e4b1b69eefa2-service-ca-bundle\") pod \"router-default-5cfd77cbdf-kcvgr\" (UID: \"a8643bdf-eeca-4201-8d47-e4b1b69eefa2\") " pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:29.156378 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.156225 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjb6d\" (UniqueName: \"kubernetes.io/projected/341fbe38-f97c-4a00-938b-6c4b9a30197e-kube-api-access-kjb6d\") pod \"console-operator-7cd56b8dbc-jlcr7\" (UID: \"341fbe38-f97c-4a00-938b-6c4b9a30197e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:29.157018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.156979 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sg8v\" (UniqueName: \"kubernetes.io/projected/a83a01b9-9670-484c-ba66-658f61d5f162-kube-api-access-9sg8v\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.157107 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.157059 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqq5\" (UniqueName: \"kubernetes.io/projected/7c0f42a0-fd35-4121-96b9-1d6363689977-kube-api-access-lhqq5\") pod \"insights-operator-7c4986d449-sbz69\" (UID: \"7c0f42a0-fd35-4121-96b9-1d6363689977\") " pod="openshift-insights/insights-operator-7c4986d449-sbz69" Oct 31 00:28:29.157158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.157132 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwh6\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\") pod \"image-registry-7c65457c94-5cwm4\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:29.158364 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.158346 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpcz\" (UniqueName: \"kubernetes.io/projected/c1caab5f-35c5-4f4d-8f81-0d18527a7ef1-kube-api-access-cwpcz\") pod \"network-check-source-6f78d59f99-cz22k\" (UID: \"c1caab5f-35c5-4f4d-8f81-0d18527a7ef1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cz22k" Oct 31 00:28:29.161008 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.160986 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjq6\" (UniqueName: \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\") pod \"periodic-gathering-bcqhp-55jrh\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:29.161047 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.161031 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tkt\" (UniqueName: \"kubernetes.io/projected/f14812ec-6d3c-4957-be87-0454c8c21d65-kube-api-access-g5tkt\") pod \"dns-default-z4v42\" (UID: \"f14812ec-6d3c-4957-be87-0454c8c21d65\") " pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:29.161154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.161134 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwt9\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.161672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.161652 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwx2\" (UniqueName: \"kubernetes.io/projected/79c0a7af-0b6e-4056-8e2d-166885cb98dc-kube-api-access-pkwx2\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.162860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.162840 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tzb\" (UniqueName: \"kubernetes.io/projected/b634720f-79f7-4d34-a971-8d1ba231b3d3-kube-api-access-m5tzb\") pod \"cluster-samples-operator-6ddb6c8b94-5h7kl\" (UID: \"b634720f-79f7-4d34-a971-8d1ba231b3d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5h7kl" Oct 31 00:28:29.165582 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.165562 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zw2\" (UniqueName: \"kubernetes.io/projected/af99842a-26b2-4e57-aae5-3aceef8f0776-kube-api-access-54zw2\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:28:29.165846 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.165829 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtjc\" (UniqueName: \"kubernetes.io/projected/9316842b-0133-456e-abeb-e0b38e7f04ff-kube-api-access-hjtjc\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.166125 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.166107 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7dm\" (UniqueName: \"kubernetes.io/projected/241ccb66-4fd3-427c-9b7f-279439013bfd-kube-api-access-wf7dm\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.166386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.166368 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sz7n\" (UniqueName: \"kubernetes.io/projected/53a51104-b6b6-4c74-a3c0-e2c5c378f28f-kube-api-access-9sz7n\") pod \"volume-data-source-validator-58c9f5f46d-86qq6\" (UID: \"53a51104-b6b6-4c74-a3c0-e2c5c378f28f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-86qq6" Oct 31 00:28:29.166441 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.166407 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj77k\" (UniqueName: \"kubernetes.io/projected/fee32755-d814-417f-8311-6a1dc5b60fa5-kube-api-access-rj77k\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.166729 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.166711 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\") pod \"image-registry-5b88d6f687-f96vv\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:29.166807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.166727 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcmw\" (UniqueName: \"kubernetes.io/projected/07e5eb4b-7a3b-43d5-81f9-cda790ace0b2-kube-api-access-mjcmw\") pod \"kas-connection-checker-685895c7c9-prfjr\" (UID: \"07e5eb4b-7a3b-43d5-81f9-cda790ace0b2\") " pod="kube-system/kas-connection-checker-685895c7c9-prfjr" Oct 31 00:28:29.166807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.166742 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2h76\" (UniqueName: \"kubernetes.io/projected/c8057563-4fad-49a2-8d82-4902eca5fa57-kube-api-access-g2h76\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:28:29.166958 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.166943 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529-kube-api-access-gmr5h\") pod \"service-ca-5fffc97bdf-427g5\" (UID: \"4b6a5917-62f5-4f02-b9eb-dcb8ac2a5529\") " pod="openshift-service-ca/service-ca-5fffc97bdf-427g5" Oct 31 00:28:29.167909 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.167893 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjvm\" (UniqueName: \"kubernetes.io/projected/38807dac-4dc9-4d27-932b-e81189808eb1-kube-api-access-lkjvm\") pod \"service-ca-operator-544d749c8c-twz2x\" (UID: \"38807dac-4dc9-4d27-932b-e81189808eb1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" Oct 31 00:28:29.168374 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.168354 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwkw\" (UniqueName: \"kubernetes.io/projected/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-kube-api-access-jnwkw\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.169401 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.169385 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsxx\" (UniqueName: \"kubernetes.io/projected/8188efe6-6d2b-46cc-ba95-7782ea3a4e6e-kube-api-access-ffsxx\") pod \"migrator-66df5d496b-89n4d\" (UID: \"8188efe6-6d2b-46cc-ba95-7782ea3a4e6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-89n4d" Oct 31 00:28:29.170399 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.170383 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fz89\" (UniqueName: \"kubernetes.io/projected/7a345f30-ddf3-478a-81e0-5d50cbf38dc4-kube-api-access-4fz89\") pod \"ingress-canary-wqdmn\" (UID: \"7a345f30-ddf3-478a-81e0-5d50cbf38dc4\") " pod="openshift-ingress-canary/ingress-canary-wqdmn" Oct 31 00:28:29.170510 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.170494 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jxj\" (UniqueName: \"kubernetes.io/projected/f4428250-46e8-4f67-bb0c-285938ee2175-kube-api-access-f8jxj\") pod \"cluster-monitoring-operator-884576c57-xlk72\" (UID: \"f4428250-46e8-4f67-bb0c-285938ee2175\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlk72" Oct 31 00:28:29.173486 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.173445 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75l4l\" (UniqueName: \"kubernetes.io/projected/be133361-4e22-4617-92fd-610a57bef9e2-kube-api-access-75l4l\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.184494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184472 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-system-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.184592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184503 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-slash\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184527 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-netd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184549 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.184592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184582 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-slash\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184583 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-socket-dir-parent\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184616 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-ovn\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184624 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184635 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-socket-dir-parent\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184639 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-sys\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184670 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-sys\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184671 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-ovn\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184585 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-netd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184669 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-kubelet\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184701 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-kubelet\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184709 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-dbus\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184729 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184585 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-system-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184754 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.184776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184774 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-bin\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184799 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-dbus\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184822 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-socket-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184840 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-kubernetes\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184853 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-cni-bin\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184857 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-lib-modules\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184874 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-conf\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184888 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-bin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184895 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-kubernetes\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184905 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184910 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-socket-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184926 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-kubelet\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184952 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-kubelet\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184960 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-bin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184954 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184981 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-netns\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185011 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-systemd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185031 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-lib-modules\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.184987 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-conf\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185041 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-systemd-units\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185054 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-run-systemd\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185065 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-sys-fs\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185070 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-run-netns\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185095 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-sys-fs\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185097 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-systemd-units\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185111 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185129 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185143 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-root\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185157 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysconfig\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185179 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-etc-kubernetes\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185198 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-root\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185205 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-run\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185220 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysconfig\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185231 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-systemd\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185238 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-etc-kubernetes\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185240 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-cni-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185244 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-run\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185255 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8057563-4fad-49a2-8d82-4902eca5fa57-host-slash\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185267 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-systemd\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185272 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-sysctl-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185280 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-netns\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185305 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-system-cni-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185306 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8057563-4fad-49a2-8d82-4902eca5fa57-host-slash\") pod \"iptables-alerter-qlblw\" (UID: \"c8057563-4fad-49a2-8d82-4902eca5fa57\") " pod="openshift-network-operator/iptables-alerter-qlblw" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185332 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-multus\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185347 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-netns\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185354 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-system-cni-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185373 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-var-lib-cni-multus\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185388 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-sys\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185353 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-sys\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.185398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185422 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-multus-certs\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185440 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-wtmp\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185456 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d172f407-b9ac-4193-b0f4-fab1c53e6874-host\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185473 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-registration-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185488 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-host\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185501 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-hostroot\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185508 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-multus-certs\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185513 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be133361-4e22-4617-92fd-610a57bef9e2-node-exporter-wtmp\") pod \"node-exporter-9wfdm\" (UID: \"be133361-4e22-4617-92fd-610a57bef9e2\") " pod="openshift-monitoring/node-exporter-9wfdm" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185530 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d172f407-b9ac-4193-b0f4-fab1c53e6874-host\") pod \"node-ca-fgvdh\" (UID: \"d172f407-b9ac-4193-b0f4-fab1c53e6874\") " pod="openshift-image-registry/node-ca-fgvdh" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185524 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-cnibin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185545 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-hostroot\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185549 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-registration-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185565 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-cnibin\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185566 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185612 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-kubelet-config\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185615 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-host\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185615 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185631 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-log-socket\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185647 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-modprobe-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185663 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185680 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-var-lib-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185694 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-node-log\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185704 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-log-socket\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185712 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af99842a-26b2-4e57-aae5-3aceef8f0776-hosts-file\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185720 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185731 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-var-lib-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185737 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-etc-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185740 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-node-log\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185721 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09934062-de9f-4e88-b25e-4f4a7947e4ce-kubelet-config\") pod \"global-pull-secret-syncer-ksxrk\" (UID: \"09934062-de9f-4e88-b25e-4f4a7947e4ce\") " pod="kube-system/global-pull-secret-syncer-ksxrk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185762 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-etc-modprobe-d\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185771 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af99842a-26b2-4e57-aae5-3aceef8f0776-hosts-file\") pod \"node-resolver-946fm\" (UID: \"af99842a-26b2-4e57-aae5-3aceef8f0776\") " pod="openshift-dns/node-resolver-946fm" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185763 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cnibin\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185778 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/799b2dd6-6d05-4ca1-bc09-79bd1634fbaa-etc-openvswitch\") pod \"ovnkube-node-c2rj2\" (UID: \"799b2dd6-6d05-4ca1-bc09-79bd1634fbaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185812 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-var-lib-kubelet\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185830 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-k8s-cni-cncf-io\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185846 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185814 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79c0a7af-0b6e-4056-8e2d-166885cb98dc-cnibin\") pod \"multus-additional-cni-plugins-zz7n4\" (UID: \"79c0a7af-0b6e-4056-8e2d-166885cb98dc\") " pod="openshift-multus/multus-additional-cni-plugins-zz7n4" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185868 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-conf-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185881 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/241ccb66-4fd3-427c-9b7f-279439013bfd-var-lib-kubelet\") pod \"tuned-mdpsp\" (UID: \"241ccb66-4fd3-427c-9b7f-279439013bfd\") " pod="openshift-cluster-node-tuning-operator/tuned-mdpsp" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185895 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83a01b9-9670-484c-ba66-658f61d5f162-crio-socket\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185915 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-device-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185935 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-etc-selinux\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185962 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-multus-conf-dir\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185981 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee32755-d814-417f-8311-6a1dc5b60fa5-device-dir\") pod \"aws-ebs-csi-driver-node-q6hlk\" (UID: \"fee32755-d814-417f-8311-6a1dc5b60fa5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q6hlk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185992 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83a01b9-9670-484c-ba66-658f61d5f162-crio-socket\") pod \"insights-runtime-extractor-jd4rk\" (UID: \"a83a01b9-9670-484c-ba66-658f61d5f162\") " pod="openshift-insights/insights-runtime-extractor-jd4rk" Oct 31 00:28:29.187104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.185896 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9316842b-0133-456e-abeb-e0b38e7f04ff-host-run-k8s-cni-cncf-io\") pod \"multus-nn2wc\" (UID: \"9316842b-0133-456e-abeb-e0b38e7f04ff\") " pod="openshift-multus/multus-nn2wc" Oct 31 00:28:29.289194 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.289174 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:28:29.426538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:29.426509 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:30.530207 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:30.530169 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:30.868513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:30.868476 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jlcr7" Oct 31 00:28:31.014953 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:31.014922 5890 generic.go:356] "Generic (PLEG): container finished" podID="604f57d5-12dc-4ade-895d-c876688423fb" containerID="15909e726f24571054749daffeee1af025c530a03ced4d10c8c161245062d74a" exitCode=0 Oct 31 00:28:31.015105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:31.015002 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" event={"ID":"604f57d5-12dc-4ade-895d-c876688423fb","Type":"ContainerDied","Data":"15909e726f24571054749daffeee1af025c530a03ced4d10c8c161245062d74a"} Oct 31 00:28:31.015391 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:31.015375 5890 scope.go:122] "RemoveContainer" containerID="15909e726f24571054749daffeee1af025c530a03ced4d10c8c161245062d74a" Oct 31 00:28:31.177890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:31.177852 5890 patch_prober.go:29] interesting pod/image-registry-5b88d6f687-f96vv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 00:28:31.178051 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:31.177906 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:31.951555 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:31.951522 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8grlc" Oct 31 00:28:32.019879 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:32.019847 5890 generic.go:356] "Generic (PLEG): container finished" podID="be133361-4e22-4617-92fd-610a57bef9e2" containerID="112d69d157e6f0d9d724fa9e71665789d513380f9c702a650a58b4b3006db67a" exitCode=0 Oct 31 00:28:32.020064 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:32.019925 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wfdm" event={"ID":"be133361-4e22-4617-92fd-610a57bef9e2","Type":"ContainerDied","Data":"112d69d157e6f0d9d724fa9e71665789d513380f9c702a650a58b4b3006db67a"} Oct 31 00:28:32.021877 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:32.021853 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-97sls" event={"ID":"604f57d5-12dc-4ade-895d-c876688423fb","Type":"ContainerStarted","Data":"ba8882a61c4c07042385e70cce6c20d96adb484610ac28587b27dfddb09de478"} Oct 31 00:28:32.908486 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:32.908453 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:32.908679 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:32.908666 5890 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:28:33.028064 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:33.028026 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wfdm" event={"ID":"be133361-4e22-4617-92fd-610a57bef9e2","Type":"ContainerStarted","Data":"0cb223e2d4f699a49d6ad42309602247704f34a1665feb3d0ac971e443889cc3"} Oct 31 00:28:33.028436 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:33.028087 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wfdm" event={"ID":"be133361-4e22-4617-92fd-610a57bef9e2","Type":"ContainerStarted","Data":"fe7fe8fc64800d896ba92a70efebf6a65eb32d484df3cd5330695a0fab29c19b"} Oct 31 00:28:33.533449 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:33.533419 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z4v42" Oct 31 00:28:34.032862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:34.032825 5890 generic.go:356] "Generic (PLEG): container finished" podID="38807dac-4dc9-4d27-932b-e81189808eb1" containerID="2e73fa5dd63035a3ea152d507ca619a2aab66c1e2f717113b8904fb4b83a68b2" exitCode=0 Oct 31 00:28:34.033286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:34.032906 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" event={"ID":"38807dac-4dc9-4d27-932b-e81189808eb1","Type":"ContainerDied","Data":"2e73fa5dd63035a3ea152d507ca619a2aab66c1e2f717113b8904fb4b83a68b2"} Oct 31 00:28:34.033286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:34.033250 5890 scope.go:122] "RemoveContainer" containerID="2e73fa5dd63035a3ea152d507ca619a2aab66c1e2f717113b8904fb4b83a68b2" Oct 31 00:28:34.034447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:34.034424 5890 generic.go:356] "Generic (PLEG): container finished" podID="7c0f42a0-fd35-4121-96b9-1d6363689977" containerID="a3f3b7e65ddb7ac0e3ec5b7e353b778534f5e8e17eea379db3f7cbb1b4118f96" exitCode=0 Oct 31 00:28:34.034543 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:34.034455 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sbz69" event={"ID":"7c0f42a0-fd35-4121-96b9-1d6363689977","Type":"ContainerDied","Data":"a3f3b7e65ddb7ac0e3ec5b7e353b778534f5e8e17eea379db3f7cbb1b4118f96"} Oct 31 00:28:34.034738 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:34.034724 5890 scope.go:122] "RemoveContainer" containerID="a3f3b7e65ddb7ac0e3ec5b7e353b778534f5e8e17eea379db3f7cbb1b4118f96" Oct 31 00:28:35.040977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:35.040946 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twz2x" event={"ID":"38807dac-4dc9-4d27-932b-e81189808eb1","Type":"ContainerStarted","Data":"a7904b0328fbda39a290d470b2934d4a0d63dea4bf8a2a1726c118dbefba2bad"} Oct 31 00:28:35.042847 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:35.042825 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sbz69" event={"ID":"7c0f42a0-fd35-4121-96b9-1d6363689977","Type":"ContainerStarted","Data":"90de3ff4187bf16db7b28c75c1cb751d54031cde15fd26c33e3ce20af13ef4a5"} Oct 31 00:28:37.017666 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:37.017620 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:28:37.315157 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:37.315044 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5cfd77cbdf-kcvgr" Oct 31 00:28:37.890547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:37.890516 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nt477" Oct 31 00:28:38.712130 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:38.712044 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b88d6f687-f96vv"] Oct 31 00:28:38.716453 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:38.716431 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:28:42.881737 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:42.881702 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2rj2" Oct 31 00:28:46.497387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.497350 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bcqhp-55jrh"] Oct 31 00:28:46.497913 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.497590 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" podUID="942dd14b-9b63-4c8a-8362-218b9043307c" containerName="insights-gathering" containerID="cri-o://22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2" gracePeriod=30 Oct 31 00:28:46.730517 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.730492 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:46.833638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.833542 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\") pod \"942dd14b-9b63-4c8a-8362-218b9043307c\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " Oct 31 00:28:46.833638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.833592 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\") pod \"942dd14b-9b63-4c8a-8362-218b9043307c\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " Oct 31 00:28:46.833638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.833625 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\" (UniqueName: \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\") pod \"942dd14b-9b63-4c8a-8362-218b9043307c\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " Oct 31 00:28:46.833958 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.833683 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\") pod \"942dd14b-9b63-4c8a-8362-218b9043307c\" (UID: \"942dd14b-9b63-4c8a-8362-218b9043307c\") " Oct 31 00:28:46.834059 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.834034 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle" pod "942dd14b-9b63-4c8a-8362-218b9043307c" (UID: "942dd14b-9b63-4c8a-8362-218b9043307c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:46.834117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.834052 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle" pod "942dd14b-9b63-4c8a-8362-218b9043307c" (UID: "942dd14b-9b63-4c8a-8362-218b9043307c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:46.834117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.834071 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path" pod "942dd14b-9b63-4c8a-8362-218b9043307c" (UID: "942dd14b-9b63-4c8a-8362-218b9043307c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:46.836116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.836027 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6" pod "942dd14b-9b63-4c8a-8362-218b9043307c" (UID: "942dd14b-9b63-4c8a-8362-218b9043307c"). InnerVolumeSpecName "kube-api-access-rvjq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:46.934363 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.934316 5890 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/942dd14b-9b63-4c8a-8362-218b9043307c-archives-path\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:46.934363 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.934358 5890 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-trusted-ca-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:46.934363 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.934368 5890 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/942dd14b-9b63-4c8a-8362-218b9043307c-service-ca-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:46.934363 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:46.934377 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjq6\" (UniqueName: \"kubernetes.io/projected/942dd14b-9b63-4c8a-8362-218b9043307c-kube-api-access-rvjq6\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:47.088642 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.088548 5890 generic.go:356] "Generic (PLEG): container finished" podID="942dd14b-9b63-4c8a-8362-218b9043307c" containerID="22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2" exitCode=2 Oct 31 00:28:47.088642 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.088610 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" event={"ID":"942dd14b-9b63-4c8a-8362-218b9043307c","Type":"ContainerDied","Data":"22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2"} Oct 31 00:28:47.088642 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.088611 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" Oct 31 00:28:47.088947 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.088649 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcqhp-55jrh" event={"ID":"942dd14b-9b63-4c8a-8362-218b9043307c","Type":"ContainerDied","Data":"c361feb9917e45dab6a40d71540b1bc98955f2846ed37b03787dad269bd5e4a4"} Oct 31 00:28:47.088947 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.088669 5890 scope.go:122] "RemoveContainer" containerID="22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2" Oct 31 00:28:47.096967 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.096947 5890 scope.go:122] "RemoveContainer" containerID="22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2" Oct 31 00:28:47.097227 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:28:47.097206 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2\": container with ID starting with 22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2 not found: ID does not exist" containerID="22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2" Oct 31 00:28:47.097276 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.097238 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2"} err="failed to get container status \"22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2\": rpc error: code = NotFound desc = could not find container \"22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2\": container with ID starting with 22081860b54d7ab1c6c57615e2f219f6de9ab3f25e87c4a9d9437478b65284d2 not found: ID does not exist" Oct 31 00:28:47.253952 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.253915 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bcqhp-55jrh"] Oct 31 00:28:47.301046 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.301012 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-bcqhp-55jrh"] Oct 31 00:28:47.930368 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:47.930333 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942dd14b-9b63-4c8a-8362-218b9043307c" path="/var/lib/kubelet/pods/942dd14b-9b63-4c8a-8362-218b9043307c/volumes" Oct 31 00:28:53.990513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:53.990466 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" podUID="8fcf1ba8-6176-418b-8477-6f5b89a16a41" containerName="registry" containerID="cri-o://a83508bb60251551009e491fc2ad8bbba7e1cd662d6d2f99222e59accfe4127b" gracePeriod=30 Oct 31 00:28:54.116542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.116505 5890 generic.go:356] "Generic (PLEG): container finished" podID="8fcf1ba8-6176-418b-8477-6f5b89a16a41" containerID="a83508bb60251551009e491fc2ad8bbba7e1cd662d6d2f99222e59accfe4127b" exitCode=0 Oct 31 00:28:54.116690 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.116559 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" event={"ID":"8fcf1ba8-6176-418b-8477-6f5b89a16a41","Type":"ContainerDied","Data":"a83508bb60251551009e491fc2ad8bbba7e1cd662d6d2f99222e59accfe4127b"} Oct 31 00:28:54.237447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.237422 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:54.299146 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.299073 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.299146 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.299109 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.299319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.299168 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.299319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.299213 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.299319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.299244 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.299319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.299265 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.300077 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.300007 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:54.301527 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.300267 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.301527 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.300352 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\") pod \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\" (UID: \"8fcf1ba8-6176-418b-8477-6f5b89a16a41\") " Oct 31 00:28:54.301527 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.300778 5890 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-trusted-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:54.301527 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.301223 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:54.303052 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.302454 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:54.303052 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.302648 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "kube-api-access-4rwh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:54.303230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.303078 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:54.303230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.303126 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:54.306815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.306041 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:54.308738 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.308612 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted" pod "8fcf1ba8-6176-418b-8477-6f5b89a16a41" (UID: "8fcf1ba8-6176-418b-8477-6f5b89a16a41"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:54.402096 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.402053 5890 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-bound-sa-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:54.402096 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.402090 5890 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8fcf1ba8-6176-418b-8477-6f5b89a16a41-ca-trust-extracted\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:54.402346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.402104 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rwh6\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-kube-api-access-4rwh6\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:54.402346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.402120 5890 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-installation-pull-secrets\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:54.402346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.402133 5890 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:54.402346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.402144 5890 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8fcf1ba8-6176-418b-8477-6f5b89a16a41-registry-certificates\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:54.402346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:54.402154 5890 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8fcf1ba8-6176-418b-8477-6f5b89a16a41-image-registry-private-configuration\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.121294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:55.121264 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" Oct 31 00:28:55.121294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:55.121270 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c65457c94-5cwm4" event={"ID":"8fcf1ba8-6176-418b-8477-6f5b89a16a41","Type":"ContainerDied","Data":"eb1954ed0a48f22fd4b4be080e1df08ee848d31bec6087aa462dbc74a4164f33"} Oct 31 00:28:55.121754 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:55.121326 5890 scope.go:122] "RemoveContainer" containerID="a83508bb60251551009e491fc2ad8bbba7e1cd662d6d2f99222e59accfe4127b" Oct 31 00:28:55.326046 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:55.326009 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c65457c94-5cwm4"] Oct 31 00:28:55.373895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:55.373827 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c65457c94-5cwm4"] Oct 31 00:28:55.930543 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:28:55.930509 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fcf1ba8-6176-418b-8477-6f5b89a16a41" path="/var/lib/kubelet/pods/8fcf1ba8-6176-418b-8477-6f5b89a16a41/volumes" Oct 31 00:29:03.731581 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:03.731514 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerName="registry" containerID="cri-o://de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8" gracePeriod=30 Oct 31 00:29:03.978296 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:03.978272 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:29:04.082197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082130 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.082197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082168 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.082405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082219 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.082405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082248 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.082405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082275 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.082405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082389 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.082609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082456 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.082609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.082520 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\") pod \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\" (UID: \"c998bbec-e9a8-45b9-8475-50a4e4fb2c35\") " Oct 31 00:29:04.083126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.083071 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:04.083126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.083080 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:04.085116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.085084 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:04.085218 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.085188 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:04.085268 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.085234 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "kube-api-access-clwt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:04.085340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.085323 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:04.085388 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.085366 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:04.089623 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.089555 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted" pod "c998bbec-e9a8-45b9-8475-50a4e4fb2c35" (UID: "c998bbec-e9a8-45b9-8475-50a4e4fb2c35"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:04.160719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.160689 5890 generic.go:356] "Generic (PLEG): container finished" podID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerID="de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8" exitCode=0 Oct 31 00:29:04.160870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.160731 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" Oct 31 00:29:04.160870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.160766 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" event={"ID":"c998bbec-e9a8-45b9-8475-50a4e4fb2c35","Type":"ContainerDied","Data":"de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8"} Oct 31 00:29:04.160870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.160832 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b88d6f687-f96vv" event={"ID":"c998bbec-e9a8-45b9-8475-50a4e4fb2c35","Type":"ContainerDied","Data":"3e5cb5dc59ef42b24a48817d6b722621ee3575f4914a0b609147abef28a022ff"} Oct 31 00:29:04.160870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.160853 5890 scope.go:122] "RemoveContainer" containerID="de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8" Oct 31 00:29:04.169665 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.169648 5890 scope.go:122] "RemoveContainer" containerID="de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8" Oct 31 00:29:04.169946 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:29:04.169929 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8\": container with ID starting with de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8 not found: ID does not exist" containerID="de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8" Oct 31 00:29:04.169999 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.169955 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8"} err="failed to get container status \"de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8\": rpc error: code = NotFound desc = could not find container \"de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8\": container with ID starting with de3a1bffde33be304be1826a8645b958b07519bfe25bb801b2205ee8ec03cdd8 not found: ID does not exist" Oct 31 00:29:04.183342 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183319 5890 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-certificates\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.183342 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183343 5890 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-registry-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.183480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183353 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clwt9\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-kube-api-access-clwt9\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.183480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183362 5890 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-ca-trust-extracted\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.183480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183371 5890 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-installation-pull-secrets\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.183480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183379 5890 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-bound-sa-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.183480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183390 5890 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-image-registry-private-configuration\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.183480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.183405 5890 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c998bbec-e9a8-45b9-8475-50a4e4fb2c35-trusted-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:04.300397 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.300365 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b88d6f687-f96vv"] Oct 31 00:29:04.304266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:04.304239 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b88d6f687-f96vv"] Oct 31 00:29:05.930412 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:29:05.930370 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" path="/var/lib/kubelet/pods/c998bbec-e9a8-45b9-8475-50a4e4fb2c35/volumes" Oct 31 00:29:07.018345 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:29:07.018311 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:29:37.016414 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:29:37.016371 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:30:07.017247 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:30:07.017203 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:30:37.016883 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:30:37.016798 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:31:07.017867 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:31:07.017826 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:31:37.017260 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:31:37.017220 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:32:07.019400 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:32:07.019356 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:32:27.577445 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577412 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h"] Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577710 5890 cpu_manager.go:414] "containerMap: removing container" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerName="registry" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577720 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerName="registry" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577731 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8fcf1ba8-6176-418b-8477-6f5b89a16a41" containerName="registry" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577736 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcf1ba8-6176-418b-8477-6f5b89a16a41" containerName="registry" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577743 5890 cpu_manager.go:414] "containerMap: removing container" podUID="942dd14b-9b63-4c8a-8362-218b9043307c" containerName="insights-gathering" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577748 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942dd14b-9b63-4c8a-8362-218b9043307c" containerName="insights-gathering" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577822 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="c998bbec-e9a8-45b9-8475-50a4e4fb2c35" containerName="registry" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577830 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcf1ba8-6176-418b-8477-6f5b89a16a41" containerName="registry" Oct 31 00:32:27.579892 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.577836 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="942dd14b-9b63-4c8a-8362-218b9043307c" containerName="insights-gathering" Oct 31 00:32:27.581116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.581100 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.590277 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.590250 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.590393 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.590296 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwghd\" (UniqueName: \"kubernetes.io/projected/45396bf8-44f0-4e3f-9398-323733e3d6fd-kube-api-access-cwghd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.590438 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.590414 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.678677 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.678648 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-2tr7z\"" Oct 31 00:32:27.678866 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.678709 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:32:27.678866 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.678732 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:32:27.679474 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.679452 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h"] Oct 31 00:32:27.691249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.691225 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.691332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.691276 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.691332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.691310 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwghd\" (UniqueName: \"kubernetes.io/projected/45396bf8-44f0-4e3f-9398-323733e3d6fd-kube-api-access-cwghd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.691614 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.691594 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.691680 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.691634 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.748094 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.748058 5890 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:32:27.750381 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.750357 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwghd\" (UniqueName: \"kubernetes.io/projected/45396bf8-44f0-4e3f-9398-323733e3d6fd-kube-api-access-cwghd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.863740 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.863670 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph"] Oct 31 00:32:27.867154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.867132 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.893670 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.893641 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.893767 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.893673 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7x4\" (UniqueName: \"kubernetes.io/projected/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-kube-api-access-vc7x4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.893767 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.893699 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.917030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.917004 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-jcxbd\"" Oct 31 00:32:27.922837 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.922749 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph"] Oct 31 00:32:27.944748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.944720 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-2tr7z\"" Oct 31 00:32:27.945754 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.945738 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:32:27.975773 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.975747 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp"] Oct 31 00:32:27.980120 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.980098 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.995187 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.995246 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7x4\" (UniqueName: \"kubernetes.io/projected/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-kube-api-access-vc7x4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.995297 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.995337 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.995399 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.995490 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcpq\" (UniqueName: \"kubernetes.io/projected/89aaf0da-b6ed-484e-ae91-60a5687dd28a-kube-api-access-lkcpq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.996610 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:27.998179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:27.997109 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:28.030294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.030265 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kkzds\"" Oct 31 00:32:28.035024 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.034915 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp"] Oct 31 00:32:28.055125 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.055099 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7x4\" (UniqueName: \"kubernetes.io/projected/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-kube-api-access-vc7x4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:28.066602 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.066578 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl"] Oct 31 00:32:28.070159 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.070139 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.095930 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.095911 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:28.096023 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.095937 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:28.096023 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.095962 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.096023 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.095996 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcpq\" (UniqueName: \"kubernetes.io/projected/89aaf0da-b6ed-484e-ae91-60a5687dd28a-kube-api-access-lkcpq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:28.096121 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.096098 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnsb\" (UniqueName: \"kubernetes.io/projected/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-kube-api-access-qlnsb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.096173 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.096154 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.096288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.096269 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:28.096324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.096310 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:28.124891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.124831 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl"] Oct 31 00:32:28.130848 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.130826 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fdghh\"" Oct 31 00:32:28.156451 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.156416 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcpq\" (UniqueName: \"kubernetes.io/projected/89aaf0da-b6ed-484e-ae91-60a5687dd28a-kube-api-access-lkcpq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:28.168998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.168972 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl"] Oct 31 00:32:28.172493 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.172477 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.182753 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.182736 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:28.196876 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.196855 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnsb\" (UniqueName: \"kubernetes.io/projected/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-kube-api-access-qlnsb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.196976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.196890 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.196976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.196922 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.196976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.196947 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.196976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.196969 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.197139 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.197070 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85k2j\" (UniqueName: \"kubernetes.io/projected/c1a62da8-5351-424d-a56d-e82a43eea0cb-kube-api-access-85k2j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.197258 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.197239 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.197326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.197306 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.228629 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.228592 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl"] Oct 31 00:32:28.232326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.231937 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-ph6nz\"" Oct 31 00:32:28.258638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.258605 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnsb\" (UniqueName: \"kubernetes.io/projected/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-kube-api-access-qlnsb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.297676 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.297642 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85k2j\" (UniqueName: \"kubernetes.io/projected/c1a62da8-5351-424d-a56d-e82a43eea0cb-kube-api-access-85k2j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.297859 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.297718 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.297859 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.297744 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.298094 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.298074 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.298195 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.298152 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.301303 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.301283 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:28.320083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.320063 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h"] Oct 31 00:32:28.322488 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:32:28.322463 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45396bf8_44f0_4e3f_9398_323733e3d6fd.slice/crio-4e23a0fea00bab5e51275fd2e91aec469b77f0969a483e14c13fbd5a08fab19b WatchSource:0}: Error finding container 4e23a0fea00bab5e51275fd2e91aec469b77f0969a483e14c13fbd5a08fab19b: Status 404 returned error can't find the container with id 4e23a0fea00bab5e51275fd2e91aec469b77f0969a483e14c13fbd5a08fab19b Oct 31 00:32:28.357661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.357628 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85k2j\" (UniqueName: \"kubernetes.io/projected/c1a62da8-5351-424d-a56d-e82a43eea0cb-kube-api-access-85k2j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.387003 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.386949 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:28.489081 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.489047 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:28.549206 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.549179 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph"] Oct 31 00:32:28.550687 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:32:28.550649 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe9c3c4_1002_4474_9084_e0bdfe2a4e44.slice/crio-1a61d549c006c2187d177fd236c75ea1f0d5e7000897f9e10785c13efa26d1f0 WatchSource:0}: Error finding container 1a61d549c006c2187d177fd236c75ea1f0d5e7000897f9e10785c13efa26d1f0: Status 404 returned error can't find the container with id 1a61d549c006c2187d177fd236c75ea1f0d5e7000897f9e10785c13efa26d1f0 Oct 31 00:32:28.665985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.665963 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp"] Oct 31 00:32:28.668909 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:32:28.668882 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89aaf0da_b6ed_484e_ae91_60a5687dd28a.slice/crio-91b1adb003a066447fcb860f064911b21a22b5106020925530f450f8218aba31 WatchSource:0}: Error finding container 91b1adb003a066447fcb860f064911b21a22b5106020925530f450f8218aba31: Status 404 returned error can't find the container with id 91b1adb003a066447fcb860f064911b21a22b5106020925530f450f8218aba31 Oct 31 00:32:28.748606 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.748578 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl"] Oct 31 00:32:28.751672 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:32:28.751642 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a40ec6c_d739_40f3_bbd3_2a30b01a7ff0.slice/crio-1bff84934f03d1d8ac62ac8011e2d625cb2d31f749a844944c57901036f3f516 WatchSource:0}: Error finding container 1bff84934f03d1d8ac62ac8011e2d625cb2d31f749a844944c57901036f3f516: Status 404 returned error can't find the container with id 1bff84934f03d1d8ac62ac8011e2d625cb2d31f749a844944c57901036f3f516 Oct 31 00:32:28.853630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.853598 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl"] Oct 31 00:32:28.856813 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:32:28.856772 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a62da8_5351_424d_a56d_e82a43eea0cb.slice/crio-386c69f1dbdb4ed8af7782280feac25330ce6207ae0046f334bf031e9280bbd5 WatchSource:0}: Error finding container 386c69f1dbdb4ed8af7782280feac25330ce6207ae0046f334bf031e9280bbd5: Status 404 returned error can't find the container with id 386c69f1dbdb4ed8af7782280feac25330ce6207ae0046f334bf031e9280bbd5 Oct 31 00:32:28.887233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.887201 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" event={"ID":"89aaf0da-b6ed-484e-ae91-60a5687dd28a","Type":"ContainerStarted","Data":"91b1adb003a066447fcb860f064911b21a22b5106020925530f450f8218aba31"} Oct 31 00:32:28.888364 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.888341 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" event={"ID":"45396bf8-44f0-4e3f-9398-323733e3d6fd","Type":"ContainerStarted","Data":"4e23a0fea00bab5e51275fd2e91aec469b77f0969a483e14c13fbd5a08fab19b"} Oct 31 00:32:28.889471 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.889451 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" event={"ID":"c1a62da8-5351-424d-a56d-e82a43eea0cb","Type":"ContainerStarted","Data":"386c69f1dbdb4ed8af7782280feac25330ce6207ae0046f334bf031e9280bbd5"} Oct 31 00:32:28.890525 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.890503 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" event={"ID":"afe9c3c4-1002-4474-9084-e0bdfe2a4e44","Type":"ContainerStarted","Data":"1a61d549c006c2187d177fd236c75ea1f0d5e7000897f9e10785c13efa26d1f0"} Oct 31 00:32:28.891586 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:28.891567 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" event={"ID":"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0","Type":"ContainerStarted","Data":"1bff84934f03d1d8ac62ac8011e2d625cb2d31f749a844944c57901036f3f516"} Oct 31 00:32:30.741461 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.741423 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4"] Oct 31 00:32:30.745614 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.745579 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.823272 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.823228 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.823439 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.823390 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rc8\" (UniqueName: \"kubernetes.io/projected/27e1bcda-d114-4cdb-8c29-969d8575ea39-kube-api-access-v2rc8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.823439 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.823427 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.844154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.844118 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jx24b\"" Oct 31 00:32:30.845238 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.845208 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4"] Oct 31 00:32:30.924896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.924407 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rc8\" (UniqueName: \"kubernetes.io/projected/27e1bcda-d114-4cdb-8c29-969d8575ea39-kube-api-access-v2rc8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.924896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.924456 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.924896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.924500 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.924896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.924838 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.925356 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.924916 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:30.988711 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:30.988682 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rc8\" (UniqueName: \"kubernetes.io/projected/27e1bcda-d114-4cdb-8c29-969d8575ea39-kube-api-access-v2rc8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:31.065116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.065037 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:31.443015 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.442978 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4"] Oct 31 00:32:31.446865 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:32:31.446839 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e1bcda_d114_4cdb_8c29_969d8575ea39.slice/crio-6fba972401a1a51ff4f7ab2ceb5de575cbc1ec1f401e4200c52ec4cbd829c7f0 WatchSource:0}: Error finding container 6fba972401a1a51ff4f7ab2ceb5de575cbc1ec1f401e4200c52ec4cbd829c7f0: Status 404 returned error can't find the container with id 6fba972401a1a51ff4f7ab2ceb5de575cbc1ec1f401e4200c52ec4cbd829c7f0 Oct 31 00:32:31.647065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.647028 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l"] Oct 31 00:32:31.651845 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.651819 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.718704 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.716515 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:32:31.718704 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.716638 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5wx8d\"" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731624 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731667 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731704 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731738 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731763 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731816 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731849 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedae545-0e22-4793-a720-230efee85446-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731903 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnx9s\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-kube-api-access-xnx9s\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.732136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.731950 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedae545-0e22-4793-a720-230efee85446-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.761170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.761090 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l"] Oct 31 00:32:31.833166 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833098 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833189 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833230 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833265 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833290 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833555 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833331 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833555 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833360 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedae545-0e22-4793-a720-230efee85446-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833555 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833415 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx9s\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-kube-api-access-xnx9s\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833555 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833468 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedae545-0e22-4793-a720-230efee85446-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833759 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833679 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.833759 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833685 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.834008 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.833982 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.834270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.834229 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.834376 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.834326 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedae545-0e22-4793-a720-230efee85446-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.837165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.837035 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedae545-0e22-4793-a720-230efee85446-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.837165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.837118 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.896412 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.896361 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx9s\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-kube-api-access-xnx9s\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.896957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.896903 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-pqv5l\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:31.909651 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.909615 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" event={"ID":"27e1bcda-d114-4cdb-8c29-969d8575ea39","Type":"ContainerStarted","Data":"6fba972401a1a51ff4f7ab2ceb5de575cbc1ec1f401e4200c52ec4cbd829c7f0"} Oct 31 00:32:31.951104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.950353 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj"] Oct 31 00:32:31.955028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.955003 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:31.970640 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:31.970587 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:32.007115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.007084 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7jxt4\"" Oct 31 00:32:32.017457 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.017424 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj"] Oct 31 00:32:32.035338 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035302 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.035484 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035437 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.035536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035521 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.035582 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035564 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.035624 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035611 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.035682 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035668 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.035751 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035734 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5k8\" (UniqueName: \"kubernetes.io/projected/2bbea852-9190-4206-a458-0ef7f9aa996f-kube-api-access-8k5k8\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.035843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.035814 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbea852-9190-4206-a458-0ef7f9aa996f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.038613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.038586 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.139986 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140226 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140041 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140226 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140077 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140413 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5k8\" (UniqueName: \"kubernetes.io/projected/2bbea852-9190-4206-a458-0ef7f9aa996f-kube-api-access-8k5k8\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140460 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140480 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbea852-9190-4206-a458-0ef7f9aa996f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140549 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140746 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140583 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140746 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140629 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140746 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140675 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.140959 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.140813 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.141284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.141256 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bbea852-9190-4206-a458-0ef7f9aa996f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.141444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.141420 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.141444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.141435 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.142771 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.142740 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.143624 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.143599 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.200101 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.200044 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bbea852-9190-4206-a458-0ef7f9aa996f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.200250 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.200234 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5k8\" (UniqueName: \"kubernetes.io/projected/2bbea852-9190-4206-a458-0ef7f9aa996f-kube-api-access-8k5k8\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj\" (UID: \"2bbea852-9190-4206-a458-0ef7f9aa996f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.271194 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.271119 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:32.364615 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.364010 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l"] Oct 31 00:32:32.369231 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:32:32.368538 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedae545_0e22_4793_a720_230efee85446.slice/crio-6e113b755dadfd1038211d9e31a429635f2e3b4ef827c6a30436e95b950ab0c6 WatchSource:0}: Error finding container 6e113b755dadfd1038211d9e31a429635f2e3b4ef827c6a30436e95b950ab0c6: Status 404 returned error can't find the container with id 6e113b755dadfd1038211d9e31a429635f2e3b4ef827c6a30436e95b950ab0c6 Oct 31 00:32:32.660613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.659571 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj"] Oct 31 00:32:32.918636 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.918475 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" event={"ID":"aedae545-0e22-4793-a720-230efee85446","Type":"ContainerStarted","Data":"6e113b755dadfd1038211d9e31a429635f2e3b4ef827c6a30436e95b950ab0c6"} Oct 31 00:32:32.923674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:32.923589 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" event={"ID":"2bbea852-9190-4206-a458-0ef7f9aa996f","Type":"ContainerStarted","Data":"8286f4a27b642fc0571b636817d45ee94851f6fcdfb1c945c50e33f8a7ef81bb"} Oct 31 00:32:37.069306 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:32:37.069217 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:32:40.128731 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:40.128692 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:40.129150 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:40.128805 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:40.129150 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:40.128855 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:40.999083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:40.999041 5890 generic.go:356] "Generic (PLEG): container finished" podID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerID="b3b91c59332d1a5e00be398dc81ec60c64cf74313e207bcc96c16f867aad08fe" exitCode=0 Oct 31 00:32:40.999265 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:40.999147 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" event={"ID":"27e1bcda-d114-4cdb-8c29-969d8575ea39","Type":"ContainerDied","Data":"b3b91c59332d1a5e00be398dc81ec60c64cf74313e207bcc96c16f867aad08fe"} Oct 31 00:32:41.003262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.003206 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" event={"ID":"2bbea852-9190-4206-a458-0ef7f9aa996f","Type":"ContainerStarted","Data":"29f193b4a57e7d8cb468bdb4ed633cfe90713f4323d98512e29e1ce380231734"} Oct 31 00:32:41.006284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.006253 5890 generic.go:356] "Generic (PLEG): container finished" podID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerID="8472ff164ed164af77b8f438b8623ccd093fda5e28a4a73484e792381628271a" exitCode=0 Oct 31 00:32:41.006385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.006339 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" event={"ID":"89aaf0da-b6ed-484e-ae91-60a5687dd28a","Type":"ContainerDied","Data":"8472ff164ed164af77b8f438b8623ccd093fda5e28a4a73484e792381628271a"} Oct 31 00:32:41.028400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.027907 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:41.028400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.027999 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:41.028400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.028050 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:41.271522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.271449 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:41.277397 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.277369 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:41.466665 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:41.466548 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" podStartSLOduration=3.006460464 podStartE2EDuration="10.466531337s" totalImagesPullingTime="7.460070873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:41.348407668 +0000 UTC m=+253.550658102" watchObservedRunningTime="2026-10-31 00:32:41.466531337 +0000 UTC m=+253.668781767" Oct 31 00:32:42.013868 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:42.013816 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" event={"ID":"aedae545-0e22-4793-a720-230efee85446","Type":"ContainerStarted","Data":"7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff"} Oct 31 00:32:42.014368 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:42.014278 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:42.015493 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:42.015468 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj" Oct 31 00:32:42.136661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:42.136603 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podStartSLOduration=2.480241805 podStartE2EDuration="11.136583767s" totalImagesPullingTime="8.656341962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:42.136435007 +0000 UTC m=+254.338685442" watchObservedRunningTime="2026-10-31 00:32:42.136583767 +0000 UTC m=+254.338834178" Oct 31 00:32:42.971299 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:42.971254 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:42.977212 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:42.977184 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:43.021977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:43.021939 5890 generic.go:356] "Generic (PLEG): container finished" podID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerID="8a12761c1c3341dffa02ce80905e67a6cedc32a0594f17416d6a786918c5acc1" exitCode=0 Oct 31 00:32:43.023068 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:43.023046 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" event={"ID":"27e1bcda-d114-4cdb-8c29-969d8575ea39","Type":"ContainerDied","Data":"8a12761c1c3341dffa02ce80905e67a6cedc32a0594f17416d6a786918c5acc1"} Oct 31 00:32:43.023704 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:43.023686 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:43.024726 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:43.024699 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:32:44.027898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.027819 5890 generic.go:356] "Generic (PLEG): container finished" podID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerID="43cff81ca9ea7315425e5d8ac6b5db53678343dda5c6a99ba5060a309a58db90" exitCode=0 Oct 31 00:32:44.027898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.027872 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" event={"ID":"89aaf0da-b6ed-484e-ae91-60a5687dd28a","Type":"ContainerDied","Data":"43cff81ca9ea7315425e5d8ac6b5db53678343dda5c6a99ba5060a309a58db90"} Oct 31 00:32:44.029551 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.029525 5890 generic.go:356] "Generic (PLEG): container finished" podID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerID="3721186074bee065c99d80169be47adef11fd9218920736208560b1dfe4b858d" exitCode=0 Oct 31 00:32:44.029638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.029602 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" event={"ID":"45396bf8-44f0-4e3f-9398-323733e3d6fd","Type":"ContainerDied","Data":"3721186074bee065c99d80169be47adef11fd9218920736208560b1dfe4b858d"} Oct 31 00:32:44.031281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.031248 5890 generic.go:356] "Generic (PLEG): container finished" podID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerID="9aa5597f432cb15e32e0f31a302b8626d80af665350b25e5bce4f71bf792aedd" exitCode=0 Oct 31 00:32:44.031373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.031298 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" event={"ID":"c1a62da8-5351-424d-a56d-e82a43eea0cb","Type":"ContainerDied","Data":"9aa5597f432cb15e32e0f31a302b8626d80af665350b25e5bce4f71bf792aedd"} Oct 31 00:32:44.032943 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.032924 5890 generic.go:356] "Generic (PLEG): container finished" podID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerID="fa65c945a96798920f8a9dca3c2f383881e8dc15d302f95d89cd23e8d8dffe6e" exitCode=0 Oct 31 00:32:44.033025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.032998 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" event={"ID":"afe9c3c4-1002-4474-9084-e0bdfe2a4e44","Type":"ContainerDied","Data":"fa65c945a96798920f8a9dca3c2f383881e8dc15d302f95d89cd23e8d8dffe6e"} Oct 31 00:32:44.034542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.034488 5890 generic.go:356] "Generic (PLEG): container finished" podID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerID="15117ccffb28b19c658b2bf4df8dd4556d86943bf40b00122db88bfb1abdf35c" exitCode=0 Oct 31 00:32:44.034542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:44.034531 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" event={"ID":"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0","Type":"ContainerDied","Data":"15117ccffb28b19c658b2bf4df8dd4556d86943bf40b00122db88bfb1abdf35c"} Oct 31 00:32:46.046658 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:46.046624 5890 generic.go:356] "Generic (PLEG): container finished" podID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerID="76815467a1b0510869c9e43c47cd7360a0ef5199ad860e0360fc0f695023a2b2" exitCode=0 Oct 31 00:32:46.047440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:46.046711 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" event={"ID":"c1a62da8-5351-424d-a56d-e82a43eea0cb","Type":"ContainerDied","Data":"76815467a1b0510869c9e43c47cd7360a0ef5199ad860e0360fc0f695023a2b2"} Oct 31 00:32:47.054681 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:47.054649 5890 generic.go:356] "Generic (PLEG): container finished" podID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerID="15dc8f0a820703bd546d53132b0253324973641d5780ba875c903c3d9506d8d4" exitCode=0 Oct 31 00:32:47.055103 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:47.054698 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" event={"ID":"afe9c3c4-1002-4474-9084-e0bdfe2a4e44","Type":"ContainerDied","Data":"15dc8f0a820703bd546d53132b0253324973641d5780ba875c903c3d9506d8d4"} Oct 31 00:32:49.072282 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:49.072248 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" event={"ID":"45396bf8-44f0-4e3f-9398-323733e3d6fd","Type":"ContainerStarted","Data":"1fb1a1cfe40c78d90d632eece328a151f129ed6652eb000e38b466a9a659e412"} Oct 31 00:32:50.080744 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:50.080698 5890 generic.go:356] "Generic (PLEG): container finished" podID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerID="1fb1a1cfe40c78d90d632eece328a151f129ed6652eb000e38b466a9a659e412" exitCode=0 Oct 31 00:32:50.081162 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:50.080860 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" event={"ID":"45396bf8-44f0-4e3f-9398-323733e3d6fd","Type":"ContainerDied","Data":"1fb1a1cfe40c78d90d632eece328a151f129ed6652eb000e38b466a9a659e412"} Oct 31 00:32:54.107320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:54.107286 5890 generic.go:356] "Generic (PLEG): container finished" podID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerID="6f8be5038b6caf5447dada525d0fded92e0b74d042bbe080e51e6fd6f83af931" exitCode=0 Oct 31 00:32:54.107832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:54.107386 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" event={"ID":"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0","Type":"ContainerDied","Data":"6f8be5038b6caf5447dada525d0fded92e0b74d042bbe080e51e6fd6f83af931"} Oct 31 00:32:54.111563 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:54.111534 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" event={"ID":"89aaf0da-b6ed-484e-ae91-60a5687dd28a","Type":"ContainerStarted","Data":"67f74f146a9df3331b02090b97ff291addb165106b6e0f8afb033e9f05435baa"} Oct 31 00:32:54.338071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:54.338005 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" podStartSLOduration=3.673370156 podStartE2EDuration="27.337983266s" totalImagesPullingTime="23.66461311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:54.33755018 +0000 UTC m=+266.539800614" watchObservedRunningTime="2026-10-31 00:32:54.337983266 +0000 UTC m=+266.540233702" Oct 31 00:32:55.119337 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:55.119177 5890 generic.go:356] "Generic (PLEG): container finished" podID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerID="67f74f146a9df3331b02090b97ff291addb165106b6e0f8afb033e9f05435baa" exitCode=0 Oct 31 00:32:55.119337 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:55.119266 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" event={"ID":"89aaf0da-b6ed-484e-ae91-60a5687dd28a","Type":"ContainerDied","Data":"67f74f146a9df3331b02090b97ff291addb165106b6e0f8afb033e9f05435baa"} Oct 31 00:32:55.122185 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:55.122142 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" event={"ID":"afe9c3c4-1002-4474-9084-e0bdfe2a4e44","Type":"ContainerStarted","Data":"77f7fdef3decc3440d935de140123a6e3cbb1cace6a1559abb54c5f96249523d"} Oct 31 00:32:55.125030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:55.125000 5890 generic.go:356] "Generic (PLEG): container finished" podID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerID="c1fd6d00e65afe62bfec5279a7d21b2b74f93e6f0d799e362cd9e71ea56b7f91" exitCode=0 Oct 31 00:32:55.125159 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:55.125048 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" event={"ID":"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0","Type":"ContainerDied","Data":"c1fd6d00e65afe62bfec5279a7d21b2b74f93e6f0d799e362cd9e71ea56b7f91"} Oct 31 00:32:55.470958 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:55.470900 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" podStartSLOduration=3.7021106120000002 podStartE2EDuration="28.470882316s" totalImagesPullingTime="24.768771704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:55.468027789 +0000 UTC m=+267.670278224" watchObservedRunningTime="2026-10-31 00:32:55.470882316 +0000 UTC m=+267.673132752" Oct 31 00:32:56.136652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.136339 5890 generic.go:356] "Generic (PLEG): container finished" podID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerID="77f7fdef3decc3440d935de140123a6e3cbb1cace6a1559abb54c5f96249523d" exitCode=0 Oct 31 00:32:56.136652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.136475 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" event={"ID":"afe9c3c4-1002-4474-9084-e0bdfe2a4e44","Type":"ContainerDied","Data":"77f7fdef3decc3440d935de140123a6e3cbb1cace6a1559abb54c5f96249523d"} Oct 31 00:32:56.139270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.139237 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" event={"ID":"27e1bcda-d114-4cdb-8c29-969d8575ea39","Type":"ContainerStarted","Data":"da163c6837ee8211c2e1041caa2a69712d5e6423f79f56c8aab58dbf6a3a94d8"} Oct 31 00:32:56.288935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.288911 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:56.318013 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.317990 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:56.365191 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.365147 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" podStartSLOduration=3.180177744 podStartE2EDuration="26.365134077s" totalImagesPullingTime="23.184956333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:56.364843759 +0000 UTC m=+268.567094189" watchObservedRunningTime="2026-10-31 00:32:56.365134077 +0000 UTC m=+268.567384511" Oct 31 00:32:56.407180 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.407150 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-util\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-util\") pod \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " Oct 31 00:32:56.407316 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.407209 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89aaf0da-b6ed-484e-ae91-60a5687dd28a-kube-api-access-lkcpq\" (UniqueName: \"kubernetes.io/projected/89aaf0da-b6ed-484e-ae91-60a5687dd28a-kube-api-access-lkcpq\") pod \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " Oct 31 00:32:56.407316 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.407295 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-bundle\") pod \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\" (UID: \"89aaf0da-b6ed-484e-ae91-60a5687dd28a\") " Oct 31 00:32:56.408135 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.408092 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-bundle" pod "89aaf0da-b6ed-484e-ae91-60a5687dd28a" (UID: "89aaf0da-b6ed-484e-ae91-60a5687dd28a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.409676 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.409649 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89aaf0da-b6ed-484e-ae91-60a5687dd28a-kube-api-access-lkcpq" pod "89aaf0da-b6ed-484e-ae91-60a5687dd28a" (UID: "89aaf0da-b6ed-484e-ae91-60a5687dd28a"). InnerVolumeSpecName "kube-api-access-lkcpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.411488 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.411442 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-util" pod "89aaf0da-b6ed-484e-ae91-60a5687dd28a" (UID: "89aaf0da-b6ed-484e-ae91-60a5687dd28a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.508548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.508515 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-kube-api-access-qlnsb\" (UniqueName: \"kubernetes.io/projected/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-kube-api-access-qlnsb\") pod \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " Oct 31 00:32:56.508688 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.508568 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-bundle\") pod \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " Oct 31 00:32:56.508688 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.508585 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-util\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-util\") pod \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\" (UID: \"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0\") " Oct 31 00:32:56.508829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.508777 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcpq\" (UniqueName: \"kubernetes.io/projected/89aaf0da-b6ed-484e-ae91-60a5687dd28a-kube-api-access-lkcpq\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.508829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.508820 5890 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.508940 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.508831 5890 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89aaf0da-b6ed-484e-ae91-60a5687dd28a-util\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.509144 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.509115 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-bundle" pod "0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" (UID: "0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.510572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.510542 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-kube-api-access-qlnsb" pod "0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" (UID: "0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0"). InnerVolumeSpecName "kube-api-access-qlnsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.513668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.513646 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-util" pod "0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" (UID: "0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.609681 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.609653 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlnsb\" (UniqueName: \"kubernetes.io/projected/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-kube-api-access-qlnsb\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.609681 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.609680 5890 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.609877 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:56.609689 5890 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0-util\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:57.145000 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.144961 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" event={"ID":"0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0","Type":"ContainerDied","Data":"1bff84934f03d1d8ac62ac8011e2d625cb2d31f749a844944c57901036f3f516"} Oct 31 00:32:57.145460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.145021 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bff84934f03d1d8ac62ac8011e2d625cb2d31f749a844944c57901036f3f516" Oct 31 00:32:57.145460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.144974 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rhvzl" Oct 31 00:32:57.147144 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.147112 5890 generic.go:356] "Generic (PLEG): container finished" podID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerID="da163c6837ee8211c2e1041caa2a69712d5e6423f79f56c8aab58dbf6a3a94d8" exitCode=0 Oct 31 00:32:57.147276 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.147205 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" event={"ID":"27e1bcda-d114-4cdb-8c29-969d8575ea39","Type":"ContainerDied","Data":"da163c6837ee8211c2e1041caa2a69712d5e6423f79f56c8aab58dbf6a3a94d8"} Oct 31 00:32:57.149083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.149062 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" Oct 31 00:32:57.149197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.149090 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avngjp" event={"ID":"89aaf0da-b6ed-484e-ae91-60a5687dd28a","Type":"ContainerDied","Data":"91b1adb003a066447fcb860f064911b21a22b5106020925530f450f8218aba31"} Oct 31 00:32:57.149197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.149127 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91b1adb003a066447fcb860f064911b21a22b5106020925530f450f8218aba31" Oct 31 00:32:57.151345 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.151320 5890 generic.go:356] "Generic (PLEG): container finished" podID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerID="a946d7953e4b4bc2061241914a873c8388a458a96fecde37fbe8c0e6bfd7ba12" exitCode=0 Oct 31 00:32:57.151443 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.151397 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" event={"ID":"c1a62da8-5351-424d-a56d-e82a43eea0cb","Type":"ContainerDied","Data":"a946d7953e4b4bc2061241914a873c8388a458a96fecde37fbe8c0e6bfd7ba12"} Oct 31 00:32:57.281360 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.281338 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:57.415481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.415446 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-bundle\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-bundle\") pod \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " Oct 31 00:32:57.415654 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.415527 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-util\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-util\") pod \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " Oct 31 00:32:57.415654 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.415577 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-kube-api-access-vc7x4\" (UniqueName: \"kubernetes.io/projected/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-kube-api-access-vc7x4\") pod \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\" (UID: \"afe9c3c4-1002-4474-9084-e0bdfe2a4e44\") " Oct 31 00:32:57.416070 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.416047 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-bundle" pod "afe9c3c4-1002-4474-9084-e0bdfe2a4e44" (UID: "afe9c3c4-1002-4474-9084-e0bdfe2a4e44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:57.417669 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.417641 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-kube-api-access-vc7x4" pod "afe9c3c4-1002-4474-9084-e0bdfe2a4e44" (UID: "afe9c3c4-1002-4474-9084-e0bdfe2a4e44"). InnerVolumeSpecName "kube-api-access-vc7x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:57.419948 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.419927 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-util" pod "afe9c3c4-1002-4474-9084-e0bdfe2a4e44" (UID: "afe9c3c4-1002-4474-9084-e0bdfe2a4e44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:57.516883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.516810 5890 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:57.516883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.516835 5890 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-util\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:57.516883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:57.516845 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7x4\" (UniqueName: \"kubernetes.io/projected/afe9c3c4-1002-4474-9084-e0bdfe2a4e44-kube-api-access-vc7x4\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:58.157092 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.157057 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" Oct 31 00:32:58.157092 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.157076 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxm8ph" event={"ID":"afe9c3c4-1002-4474-9084-e0bdfe2a4e44","Type":"ContainerDied","Data":"1a61d549c006c2187d177fd236c75ea1f0d5e7000897f9e10785c13efa26d1f0"} Oct 31 00:32:58.157568 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.157134 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a61d549c006c2187d177fd236c75ea1f0d5e7000897f9e10785c13efa26d1f0" Oct 31 00:32:58.282103 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.282077 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:58.310551 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.310523 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:58.425438 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.425409 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-bundle\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-bundle\") pod \"27e1bcda-d114-4cdb-8c29-969d8575ea39\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " Oct 31 00:32:58.425577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.425482 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e1bcda-d114-4cdb-8c29-969d8575ea39-kube-api-access-v2rc8\" (UniqueName: \"kubernetes.io/projected/27e1bcda-d114-4cdb-8c29-969d8575ea39-kube-api-access-v2rc8\") pod \"27e1bcda-d114-4cdb-8c29-969d8575ea39\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " Oct 31 00:32:58.425577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.425511 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a62da8-5351-424d-a56d-e82a43eea0cb-kube-api-access-85k2j\" (UniqueName: \"kubernetes.io/projected/c1a62da8-5351-424d-a56d-e82a43eea0cb-kube-api-access-85k2j\") pod \"c1a62da8-5351-424d-a56d-e82a43eea0cb\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " Oct 31 00:32:58.425577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.425529 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-util\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-util\") pod \"27e1bcda-d114-4cdb-8c29-969d8575ea39\" (UID: \"27e1bcda-d114-4cdb-8c29-969d8575ea39\") " Oct 31 00:32:58.425577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.425573 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-bundle\") pod \"c1a62da8-5351-424d-a56d-e82a43eea0cb\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " Oct 31 00:32:58.425766 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.425599 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-util\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-util\") pod \"c1a62da8-5351-424d-a56d-e82a43eea0cb\" (UID: \"c1a62da8-5351-424d-a56d-e82a43eea0cb\") " Oct 31 00:32:58.426185 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.426148 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-bundle" pod "c1a62da8-5351-424d-a56d-e82a43eea0cb" (UID: "c1a62da8-5351-424d-a56d-e82a43eea0cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:58.426307 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.426160 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-bundle" pod "27e1bcda-d114-4cdb-8c29-969d8575ea39" (UID: "27e1bcda-d114-4cdb-8c29-969d8575ea39"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:58.427645 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.427624 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e1bcda-d114-4cdb-8c29-969d8575ea39-kube-api-access-v2rc8" pod "27e1bcda-d114-4cdb-8c29-969d8575ea39" (UID: "27e1bcda-d114-4cdb-8c29-969d8575ea39"). InnerVolumeSpecName "kube-api-access-v2rc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:58.427754 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.427734 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a62da8-5351-424d-a56d-e82a43eea0cb-kube-api-access-85k2j" pod "c1a62da8-5351-424d-a56d-e82a43eea0cb" (UID: "c1a62da8-5351-424d-a56d-e82a43eea0cb"). InnerVolumeSpecName "kube-api-access-85k2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:58.431903 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.431876 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-util" pod "27e1bcda-d114-4cdb-8c29-969d8575ea39" (UID: "27e1bcda-d114-4cdb-8c29-969d8575ea39"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:58.432377 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.432357 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-util" pod "c1a62da8-5351-424d-a56d-e82a43eea0cb" (UID: "c1a62da8-5351-424d-a56d-e82a43eea0cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:58.526266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.526197 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2rc8\" (UniqueName: \"kubernetes.io/projected/27e1bcda-d114-4cdb-8c29-969d8575ea39-kube-api-access-v2rc8\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:58.526266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.526226 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85k2j\" (UniqueName: \"kubernetes.io/projected/c1a62da8-5351-424d-a56d-e82a43eea0cb-kube-api-access-85k2j\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:58.526266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.526236 5890 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-util\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:58.526266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.526245 5890 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:58.526266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.526254 5890 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1a62da8-5351-424d-a56d-e82a43eea0cb-util\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:58.526266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:58.526262 5890 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27e1bcda-d114-4cdb-8c29-969d8575ea39-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:59.163649 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:59.163609 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" Oct 31 00:32:59.163649 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:59.163631 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lmkjl" event={"ID":"c1a62da8-5351-424d-a56d-e82a43eea0cb","Type":"ContainerDied","Data":"386c69f1dbdb4ed8af7782280feac25330ce6207ae0046f334bf031e9280bbd5"} Oct 31 00:32:59.164246 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:59.163691 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="386c69f1dbdb4ed8af7782280feac25330ce6207ae0046f334bf031e9280bbd5" Oct 31 00:32:59.165868 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:59.165848 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" Oct 31 00:32:59.165985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:59.165843 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71czfp4" event={"ID":"27e1bcda-d114-4cdb-8c29-969d8575ea39","Type":"ContainerDied","Data":"6fba972401a1a51ff4f7ab2ceb5de575cbc1ec1f401e4200c52ec4cbd829c7f0"} Oct 31 00:32:59.165985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:32:59.165958 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fba972401a1a51ff4f7ab2ceb5de575cbc1ec1f401e4200c52ec4cbd829c7f0" Oct 31 00:33:03.186624 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.186590 5890 generic.go:356] "Generic (PLEG): container finished" podID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerID="0d5190419e5ff8b6f891187716f27ba3aa080ad514517b0f1d4f8ff87dd0768f" exitCode=0 Oct 31 00:33:03.187029 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.186651 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" event={"ID":"45396bf8-44f0-4e3f-9398-323733e3d6fd","Type":"ContainerDied","Data":"0d5190419e5ff8b6f891187716f27ba3aa080ad514517b0f1d4f8ff87dd0768f"} Oct 31 00:33:03.995754 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.995715 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb"] Oct 31 00:33:03.996037 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996024 5890 cpu_manager.go:414] "containerMap: removing container" podUID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerName="util" Oct 31 00:33:03.996084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996037 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerName="util" Oct 31 00:33:03.996084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996051 5890 cpu_manager.go:414] "containerMap: removing container" podUID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerName="util" Oct 31 00:33:03.996084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996056 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerName="util" Oct 31 00:33:03.996084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996066 5890 cpu_manager.go:414] "containerMap: removing container" podUID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerName="extract" Oct 31 00:33:03.996084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996071 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerName="extract" Oct 31 00:33:03.996084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996077 5890 cpu_manager.go:414] "containerMap: removing container" podUID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerName="extract" Oct 31 00:33:03.996084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996082 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996088 5890 cpu_manager.go:414] "containerMap: removing container" podUID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996093 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996099 5890 cpu_manager.go:414] "containerMap: removing container" podUID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996104 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996112 5890 cpu_manager.go:414] "containerMap: removing container" podUID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996117 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996127 5890 cpu_manager.go:414] "containerMap: removing container" podUID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996132 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996141 5890 cpu_manager.go:414] "containerMap: removing container" podUID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerName="util" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996146 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerName="util" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996157 5890 cpu_manager.go:414] "containerMap: removing container" podUID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996162 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996170 5890 cpu_manager.go:414] "containerMap: removing container" podUID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996175 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996182 5890 cpu_manager.go:414] "containerMap: removing container" podUID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996186 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996197 5890 cpu_manager.go:414] "containerMap: removing container" podUID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerName="util" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996201 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerName="util" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996207 5890 cpu_manager.go:414] "containerMap: removing container" podUID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996211 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerName="pull" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996218 5890 cpu_manager.go:414] "containerMap: removing container" podUID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerName="util" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996222 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerName="util" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996270 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e1bcda-d114-4cdb-8c29-969d8575ea39" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996278 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="89aaf0da-b6ed-484e-ae91-60a5687dd28a" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996287 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a40ec6c-d739-40f3-bbd3-2a30b01a7ff0" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996292 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe9c3c4-1002-4474-9084-e0bdfe2a4e44" containerName="extract" Oct 31 00:33:03.996284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.996299 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a62da8-5351-424d-a56d-e82a43eea0cb" containerName="extract" Oct 31 00:33:03.998316 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:03.998300 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.068386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.068350 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.068550 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.068400 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-tmp\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.068550 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.068488 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-manifests\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.068550 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.068522 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5sc9\" (UniqueName: \"kubernetes.io/projected/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-kube-api-access-h5sc9\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.068550 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.068547 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.068746 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.068566 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.079613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.079585 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-km8d9\"" Oct 31 00:33:04.079858 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.079844 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:33:04.079935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.079872 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:33:04.079935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.079905 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:33:04.080946 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.080927 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:33:04.081683 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.081667 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:33:04.160623 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.160592 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb"] Oct 31 00:33:04.169020 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.168977 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-tmp\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.169143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.169030 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-manifests\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.169143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.169069 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5sc9\" (UniqueName: \"kubernetes.io/projected/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-kube-api-access-h5sc9\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.169143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.169090 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.169143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.169119 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.169372 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.169171 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.169501 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.169479 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-manifests\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.171565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.171536 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.171676 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.171601 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.171735 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.171674 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-tmp\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.172196 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.172171 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-metrics-cert\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.232680 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.232643 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5sc9\" (UniqueName: \"kubernetes.io/projected/e96d1c67-05bb-4477-b47e-9ea18be4e3e8-kube-api-access-h5sc9\") pod \"opendatahub-operator-controller-manager-c54754479-qtnzb\" (UID: \"e96d1c67-05bb-4477-b47e-9ea18be4e3e8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.321409 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.321384 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:04.321774 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.321744 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:33:04.371285 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.371155 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-util\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-util\") pod \"45396bf8-44f0-4e3f-9398-323733e3d6fd\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " Oct 31 00:33:04.371285 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.371243 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45396bf8-44f0-4e3f-9398-323733e3d6fd-kube-api-access-cwghd\" (UniqueName: \"kubernetes.io/projected/45396bf8-44f0-4e3f-9398-323733e3d6fd-kube-api-access-cwghd\") pod \"45396bf8-44f0-4e3f-9398-323733e3d6fd\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " Oct 31 00:33:04.371285 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.371279 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-bundle\") pod \"45396bf8-44f0-4e3f-9398-323733e3d6fd\" (UID: \"45396bf8-44f0-4e3f-9398-323733e3d6fd\") " Oct 31 00:33:04.372115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.372010 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-bundle" pod "45396bf8-44f0-4e3f-9398-323733e3d6fd" (UID: "45396bf8-44f0-4e3f-9398-323733e3d6fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:04.376183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.376160 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45396bf8-44f0-4e3f-9398-323733e3d6fd-kube-api-access-cwghd" pod "45396bf8-44f0-4e3f-9398-323733e3d6fd" (UID: "45396bf8-44f0-4e3f-9398-323733e3d6fd"). InnerVolumeSpecName "kube-api-access-cwghd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:04.379992 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.379960 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-util" pod "45396bf8-44f0-4e3f-9398-323733e3d6fd" (UID: "45396bf8-44f0-4e3f-9398-323733e3d6fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:04.472256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.472221 5890 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-util\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:04.472256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.472248 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwghd\" (UniqueName: \"kubernetes.io/projected/45396bf8-44f0-4e3f-9398-323733e3d6fd-kube-api-access-cwghd\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:04.472256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.472258 5890 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45396bf8-44f0-4e3f-9398-323733e3d6fd-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:04.798097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:04.798064 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb"] Oct 31 00:33:04.802197 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:04.802170 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96d1c67_05bb_4477_b47e_9ea18be4e3e8.slice/crio-10f0a0970597cda9ba43e48c9b1f94b7b40a97efbf65d6027855c0b979cdabc3 WatchSource:0}: Error finding container 10f0a0970597cda9ba43e48c9b1f94b7b40a97efbf65d6027855c0b979cdabc3: Status 404 returned error can't find the container with id 10f0a0970597cda9ba43e48c9b1f94b7b40a97efbf65d6027855c0b979cdabc3 Oct 31 00:33:05.197538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:05.197502 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" event={"ID":"e96d1c67-05bb-4477-b47e-9ea18be4e3e8","Type":"ContainerStarted","Data":"10f0a0970597cda9ba43e48c9b1f94b7b40a97efbf65d6027855c0b979cdabc3"} Oct 31 00:33:05.199398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:05.199365 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" event={"ID":"45396bf8-44f0-4e3f-9398-323733e3d6fd","Type":"ContainerDied","Data":"4e23a0fea00bab5e51275fd2e91aec469b77f0969a483e14c13fbd5a08fab19b"} Oct 31 00:33:05.199540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:05.199406 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e23a0fea00bab5e51275fd2e91aec469b77f0969a483e14c13fbd5a08fab19b" Oct 31 00:33:05.199540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:05.199415 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448pp6h" Oct 31 00:33:07.021538 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:33:07.021494 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:33:09.219497 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:09.219392 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" event={"ID":"e96d1c67-05bb-4477-b47e-9ea18be4e3e8","Type":"ContainerStarted","Data":"e3526594f42ace74f28b4df1b3acdd32e2baacf27c8a3f7bfc29d189cd7e2450"} Oct 31 00:33:10.225143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:10.225110 5890 generic.go:356] "Generic (PLEG): container finished" podID="e96d1c67-05bb-4477-b47e-9ea18be4e3e8" containerID="e3526594f42ace74f28b4df1b3acdd32e2baacf27c8a3f7bfc29d189cd7e2450" exitCode=0 Oct 31 00:33:10.225545 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:10.225183 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" event={"ID":"e96d1c67-05bb-4477-b47e-9ea18be4e3e8","Type":"ContainerDied","Data":"e3526594f42ace74f28b4df1b3acdd32e2baacf27c8a3f7bfc29d189cd7e2450"} Oct 31 00:33:11.231824 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:11.231764 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" event={"ID":"e96d1c67-05bb-4477-b47e-9ea18be4e3e8","Type":"ContainerStarted","Data":"21d3ef3c312af89011fd74ff33a606dcdc54dc154187ba4e6dc7145315571a00"} Oct 31 00:33:11.232199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:11.231864 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:11.391144 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:11.391090 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" podStartSLOduration=2.875829953 podStartE2EDuration="8.391077099s" totalImagesPullingTime="5.050669301s" totalInitContainerRuntime="464.577845ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:11.389938178 +0000 UTC m=+283.592188608" watchObservedRunningTime="2026-10-31 00:33:11.391077099 +0000 UTC m=+283.593327523" Oct 31 00:33:12.444230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444187 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl"] Oct 31 00:33:12.444707 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444686 5890 cpu_manager.go:414] "containerMap: removing container" podUID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerName="util" Oct 31 00:33:12.444825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444710 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerName="util" Oct 31 00:33:12.444825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444749 5890 cpu_manager.go:414] "containerMap: removing container" podUID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerName="pull" Oct 31 00:33:12.444825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444757 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerName="pull" Oct 31 00:33:12.444825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444767 5890 cpu_manager.go:414] "containerMap: removing container" podUID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerName="extract" Oct 31 00:33:12.444825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444774 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerName="extract" Oct 31 00:33:12.445075 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.444898 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="45396bf8-44f0-4e3f-9398-323733e3d6fd" containerName="extract" Oct 31 00:33:12.446975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.446951 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" Oct 31 00:33:12.511691 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.510981 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-brsj9\"" Oct 31 00:33:12.519598 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.519564 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl"] Oct 31 00:33:12.539161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.539131 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8gk9\" (UniqueName: \"kubernetes.io/projected/f877144e-05ba-4ead-a809-5daea44d1075-kube-api-access-h8gk9\") pod \"authorino-operator-7b7bb4bc77-wqmtl\" (UID: \"f877144e-05ba-4ead-a809-5daea44d1075\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" Oct 31 00:33:12.570261 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.570231 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:33:12.570401 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.570318 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:33:12.640349 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.640318 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gk9\" (UniqueName: \"kubernetes.io/projected/f877144e-05ba-4ead-a809-5daea44d1075-kube-api-access-h8gk9\") pod \"authorino-operator-7b7bb4bc77-wqmtl\" (UID: \"f877144e-05ba-4ead-a809-5daea44d1075\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" Oct 31 00:33:12.700679 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.700607 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gk9\" (UniqueName: \"kubernetes.io/projected/f877144e-05ba-4ead-a809-5daea44d1075-kube-api-access-h8gk9\") pod \"authorino-operator-7b7bb4bc77-wqmtl\" (UID: \"f877144e-05ba-4ead-a809-5daea44d1075\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" Oct 31 00:33:12.759647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:12.759611 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" Oct 31 00:33:13.138458 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:13.138364 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl"] Oct 31 00:33:13.141392 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:13.141362 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf877144e_05ba_4ead_a809_5daea44d1075.slice/crio-5cacb9cd5b1b5c7abca8dcd7a15d018ef34adf85eacda758357740fa5b646554 WatchSource:0}: Error finding container 5cacb9cd5b1b5c7abca8dcd7a15d018ef34adf85eacda758357740fa5b646554: Status 404 returned error can't find the container with id 5cacb9cd5b1b5c7abca8dcd7a15d018ef34adf85eacda758357740fa5b646554 Oct 31 00:33:13.241960 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:13.241926 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" event={"ID":"f877144e-05ba-4ead-a809-5daea44d1075","Type":"ContainerStarted","Data":"5cacb9cd5b1b5c7abca8dcd7a15d018ef34adf85eacda758357740fa5b646554"} Oct 31 00:33:16.430239 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.430204 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4"] Oct 31 00:33:16.433697 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.433680 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:16.471166 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.471130 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a26720e7-7494-4425-b6df-cdc7bdf82c1f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-grjd4\" (UID: \"a26720e7-7494-4425-b6df-cdc7bdf82c1f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:16.471308 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.471170 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79zw\" (UniqueName: \"kubernetes.io/projected/a26720e7-7494-4425-b6df-cdc7bdf82c1f-kube-api-access-v79zw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-grjd4\" (UID: \"a26720e7-7494-4425-b6df-cdc7bdf82c1f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:16.478036 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.478012 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4"] Oct 31 00:33:16.524236 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.524203 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 00:33:16.524374 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.524291 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:33:16.524374 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.524307 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-l4jbm\"" Oct 31 00:33:16.571915 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.571886 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a26720e7-7494-4425-b6df-cdc7bdf82c1f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-grjd4\" (UID: \"a26720e7-7494-4425-b6df-cdc7bdf82c1f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:16.571915 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.571919 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79zw\" (UniqueName: \"kubernetes.io/projected/a26720e7-7494-4425-b6df-cdc7bdf82c1f-kube-api-access-v79zw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-grjd4\" (UID: \"a26720e7-7494-4425-b6df-cdc7bdf82c1f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:16.572408 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.572390 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a26720e7-7494-4425-b6df-cdc7bdf82c1f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-grjd4\" (UID: \"a26720e7-7494-4425-b6df-cdc7bdf82c1f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:16.643432 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.643403 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79zw\" (UniqueName: \"kubernetes.io/projected/a26720e7-7494-4425-b6df-cdc7bdf82c1f-kube-api-access-v79zw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-grjd4\" (UID: \"a26720e7-7494-4425-b6df-cdc7bdf82c1f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:16.745176 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:16.745103 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" Oct 31 00:33:17.214534 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:17.214509 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4"] Oct 31 00:33:17.217217 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:17.217189 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26720e7_7494_4425_b6df_cdc7bdf82c1f.slice/crio-66f5477f43c685a684ddfb73befcfe2c67c8616500b73af58da713a93e5e48c9 WatchSource:0}: Error finding container 66f5477f43c685a684ddfb73befcfe2c67c8616500b73af58da713a93e5e48c9: Status 404 returned error can't find the container with id 66f5477f43c685a684ddfb73befcfe2c67c8616500b73af58da713a93e5e48c9 Oct 31 00:33:17.264426 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:17.264392 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" event={"ID":"a26720e7-7494-4425-b6df-cdc7bdf82c1f","Type":"ContainerStarted","Data":"66f5477f43c685a684ddfb73befcfe2c67c8616500b73af58da713a93e5e48c9"} Oct 31 00:33:19.276007 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:19.275967 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" event={"ID":"f877144e-05ba-4ead-a809-5daea44d1075","Type":"ContainerStarted","Data":"1a98499c6841675e0ebd7df5a891a351eac7f659e960f18d3157b9617a540f99"} Oct 31 00:33:19.276395 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:19.276069 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" Oct 31 00:33:19.388368 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:19.388314 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" podStartSLOduration=2.169245063 podStartE2EDuration="7.388300652s" totalImagesPullingTime="5.219055589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:19.388074382 +0000 UTC m=+291.590324815" watchObservedRunningTime="2026-10-31 00:33:19.388300652 +0000 UTC m=+291.590551085" Oct 31 00:33:21.053650 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.053613 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn"] Oct 31 00:33:21.055964 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.055934 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.104199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.104172 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn"] Oct 31 00:33:21.108536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.108512 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ffwpz\"" Oct 31 00:33:21.110752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.110724 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0c2c-a45a-45d0-9e71-505ad5dff953-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6b4pn\" (UID: \"8c5a0c2c-a45a-45d0-9e71-505ad5dff953\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.110973 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.110802 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86s56\" (UniqueName: \"kubernetes.io/projected/8c5a0c2c-a45a-45d0-9e71-505ad5dff953-kube-api-access-86s56\") pod \"kuadrant-operator-controller-manager-678d87766f-6b4pn\" (UID: \"8c5a0c2c-a45a-45d0-9e71-505ad5dff953\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.211944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.211909 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0c2c-a45a-45d0-9e71-505ad5dff953-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6b4pn\" (UID: \"8c5a0c2c-a45a-45d0-9e71-505ad5dff953\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.212099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.211955 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86s56\" (UniqueName: \"kubernetes.io/projected/8c5a0c2c-a45a-45d0-9e71-505ad5dff953-kube-api-access-86s56\") pod \"kuadrant-operator-controller-manager-678d87766f-6b4pn\" (UID: \"8c5a0c2c-a45a-45d0-9e71-505ad5dff953\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.212440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.212419 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0c2c-a45a-45d0-9e71-505ad5dff953-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6b4pn\" (UID: \"8c5a0c2c-a45a-45d0-9e71-505ad5dff953\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.284581 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.284552 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86s56\" (UniqueName: \"kubernetes.io/projected/8c5a0c2c-a45a-45d0-9e71-505ad5dff953-kube-api-access-86s56\") pod \"kuadrant-operator-controller-manager-678d87766f-6b4pn\" (UID: \"8c5a0c2c-a45a-45d0-9e71-505ad5dff953\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.369093 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.369004 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:21.846890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:21.846860 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn"] Oct 31 00:33:21.849901 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:21.849873 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5a0c2c_a45a_45d0_9e71_505ad5dff953.slice/crio-11de55c8914f84ed14106996689e297a9636fc9e133cd739584c4d7bb94884ea WatchSource:0}: Error finding container 11de55c8914f84ed14106996689e297a9636fc9e133cd739584c4d7bb94884ea: Status 404 returned error can't find the container with id 11de55c8914f84ed14106996689e297a9636fc9e133cd739584c4d7bb94884ea Oct 31 00:33:22.238535 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:22.238502 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-qtnzb" Oct 31 00:33:22.292274 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:22.292239 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" event={"ID":"8c5a0c2c-a45a-45d0-9e71-505ad5dff953","Type":"ContainerStarted","Data":"11de55c8914f84ed14106996689e297a9636fc9e133cd739584c4d7bb94884ea"} Oct 31 00:33:23.299374 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:23.299311 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" event={"ID":"a26720e7-7494-4425-b6df-cdc7bdf82c1f","Type":"ContainerStarted","Data":"2869f55e6c43fcfcb0eadfebcc4e6be8f7669fffe769bcce138aaa5a08d6f84f"} Oct 31 00:33:25.199588 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.199530 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-grjd4" podStartSLOduration=3.687202913 podStartE2EDuration="9.199514002s" totalImagesPullingTime="5.512311089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:23.425926564 +0000 UTC m=+295.628176996" watchObservedRunningTime="2026-10-31 00:33:25.199514002 +0000 UTC m=+297.401764442" Oct 31 00:33:25.200012 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.199989 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb"] Oct 31 00:33:25.222968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.222933 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" Oct 31 00:33:25.273923 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.273893 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb"] Oct 31 00:33:25.319651 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.319617 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-b8zl2\"" Oct 31 00:33:25.348098 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.348059 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gv8f\" (UniqueName: \"kubernetes.io/projected/1fd906c8-7686-4cb4-9057-0d8b1da239d1-kube-api-access-2gv8f\") pod \"limitador-operator-controller-manager-5778bb4449-6k4fb\" (UID: \"1fd906c8-7686-4cb4-9057-0d8b1da239d1\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" Oct 31 00:33:25.449161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.449119 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gv8f\" (UniqueName: \"kubernetes.io/projected/1fd906c8-7686-4cb4-9057-0d8b1da239d1-kube-api-access-2gv8f\") pod \"limitador-operator-controller-manager-5778bb4449-6k4fb\" (UID: \"1fd906c8-7686-4cb4-9057-0d8b1da239d1\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" Oct 31 00:33:25.513557 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.513485 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gv8f\" (UniqueName: \"kubernetes.io/projected/1fd906c8-7686-4cb4-9057-0d8b1da239d1-kube-api-access-2gv8f\") pod \"limitador-operator-controller-manager-5778bb4449-6k4fb\" (UID: \"1fd906c8-7686-4cb4-9057-0d8b1da239d1\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" Oct 31 00:33:25.549011 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.548980 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" Oct 31 00:33:25.980170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:25.980145 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb"] Oct 31 00:33:25.982954 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:25.982911 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd906c8_7686_4cb4_9057_0d8b1da239d1.slice/crio-9777890ce873c917f0095a9c909a9815d1b639f74b02243b57c0946394e3318f WatchSource:0}: Error finding container 9777890ce873c917f0095a9c909a9815d1b639f74b02243b57c0946394e3318f: Status 404 returned error can't find the container with id 9777890ce873c917f0095a9c909a9815d1b639f74b02243b57c0946394e3318f Oct 31 00:33:26.315942 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:26.315863 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" event={"ID":"1fd906c8-7686-4cb4-9057-0d8b1da239d1","Type":"ContainerStarted","Data":"9777890ce873c917f0095a9c909a9815d1b639f74b02243b57c0946394e3318f"} Oct 31 00:33:27.955993 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:27.955943 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:33:27.957328 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:27.957300 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:33:28.329300 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.329266 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" event={"ID":"8c5a0c2c-a45a-45d0-9e71-505ad5dff953","Type":"ContainerStarted","Data":"c88d3d3ce6238e8608267c3e995737cccc522cd24258a45c0ef6c66596ffa712"} Oct 31 00:33:28.329534 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.329493 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:28.585377 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.585274 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" podStartSLOduration=2.251392622 podStartE2EDuration="8.585258046s" totalImagesPullingTime="6.333865424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:28.538910567 +0000 UTC m=+300.741161001" watchObservedRunningTime="2026-10-31 00:33:28.585258046 +0000 UTC m=+300.787508476" Oct 31 00:33:28.688596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.688556 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g"] Oct 31 00:33:28.691986 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.691962 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" Oct 31 00:33:28.754155 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.754120 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g7gdb\"" Oct 31 00:33:28.755138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.755106 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g"] Oct 31 00:33:28.778604 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.778575 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mvx\" (UniqueName: \"kubernetes.io/projected/d98fb307-575b-4e36-8e52-9398d884733a-kube-api-access-f9mvx\") pod \"dns-operator-controller-manager-8557cbdb84-zp68g\" (UID: \"d98fb307-575b-4e36-8e52-9398d884733a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" Oct 31 00:33:28.820467 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.820434 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 00:33:28.879165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.879085 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvx\" (UniqueName: \"kubernetes.io/projected/d98fb307-575b-4e36-8e52-9398d884733a-kube-api-access-f9mvx\") pod \"dns-operator-controller-manager-8557cbdb84-zp68g\" (UID: \"d98fb307-575b-4e36-8e52-9398d884733a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" Oct 31 00:33:28.954676 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:28.954649 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mvx\" (UniqueName: \"kubernetes.io/projected/d98fb307-575b-4e36-8e52-9398d884733a-kube-api-access-f9mvx\") pod \"dns-operator-controller-manager-8557cbdb84-zp68g\" (UID: \"d98fb307-575b-4e36-8e52-9398d884733a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" Oct 31 00:33:29.004647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:29.004604 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" Oct 31 00:33:29.336254 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:29.336212 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" event={"ID":"1fd906c8-7686-4cb4-9057-0d8b1da239d1","Type":"ContainerStarted","Data":"34f632dbe09a34a05925dab06d7ed7575234e078afe398ab368ab539cdcb8a76"} Oct 31 00:33:29.336424 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:29.336364 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" Oct 31 00:33:29.475693 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:29.475639 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" podStartSLOduration=1.271978225 podStartE2EDuration="4.475621865s" totalImagesPullingTime="3.20364364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:29.464599211 +0000 UTC m=+301.666849644" watchObservedRunningTime="2026-10-31 00:33:29.475621865 +0000 UTC m=+301.677872299" Oct 31 00:33:29.476482 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:29.476459 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g"] Oct 31 00:33:29.477651 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:29.477622 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98fb307_575b_4e36_8e52_9398d884733a.slice/crio-bc51b7efb0919926f72db048dbf0f2ea40127e0ca8db495784c5bb43c400e040 WatchSource:0}: Error finding container bc51b7efb0919926f72db048dbf0f2ea40127e0ca8db495784c5bb43c400e040: Status 404 returned error can't find the container with id bc51b7efb0919926f72db048dbf0f2ea40127e0ca8db495784c5bb43c400e040 Oct 31 00:33:29.480171 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:29.480156 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:33:30.282891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:30.282855 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqmtl" Oct 31 00:33:30.343767 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:30.343731 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" event={"ID":"d98fb307-575b-4e36-8e52-9398d884733a","Type":"ContainerStarted","Data":"bc51b7efb0919926f72db048dbf0f2ea40127e0ca8db495784c5bb43c400e040"} Oct 31 00:33:34.364578 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:34.364485 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" event={"ID":"d98fb307-575b-4e36-8e52-9398d884733a","Type":"ContainerStarted","Data":"7a189660f291f6f19e421c7d17950ba740125c4c9b7f9e3514e85924ac37ccae"} Oct 31 00:33:34.365021 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:34.364601 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" Oct 31 00:33:34.477281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:34.477230 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" podStartSLOduration=1.847689673 podStartE2EDuration="6.477217514s" totalImagesPullingTime="4.629527841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:34.476658965 +0000 UTC m=+306.678909398" watchObservedRunningTime="2026-10-31 00:33:34.477217514 +0000 UTC m=+306.679467947" Oct 31 00:33:37.017044 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:33:37.016999 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:33:38.438966 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.438927 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f"] Oct 31 00:33:38.444861 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.444840 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.494996 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.494965 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kr66l\"" Oct 31 00:33:38.544736 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.544696 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f"] Oct 31 00:33:38.572843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.572813 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573033 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.572860 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2g6\" (UniqueName: \"kubernetes.io/projected/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-kube-api-access-tp2g6\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573033 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.572947 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573033 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.572991 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573185 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.573059 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573185 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.573089 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573185 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.573157 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573301 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.573196 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.573301 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.573219 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.674637 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674594 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.674860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674652 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2g6\" (UniqueName: \"kubernetes.io/projected/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-kube-api-access-tp2g6\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.674860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674700 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.674860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674724 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.674860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674750 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674853 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674924 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.674977 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.675028 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675335 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.675180 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675335 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.675217 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.675360 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.675360 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.675852 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.675824 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.677078 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.677056 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.677281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.677263 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.733726 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.733657 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.733858 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.733805 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2g6\" (UniqueName: \"kubernetes.io/projected/bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699-kube-api-access-tp2g6\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fgwg4f\" (UID: \"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:38.758335 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:38.758309 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:39.230105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.230077 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f"] Oct 31 00:33:39.232314 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:39.232292 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd9901e_a5e0_47e3_8fb9_2bbfbfc05699.slice/crio-84cf3efd6278b1466041acbe0f6cf3012c52e12f13616fc0edf24b59e8ba2c85 WatchSource:0}: Error finding container 84cf3efd6278b1466041acbe0f6cf3012c52e12f13616fc0edf24b59e8ba2c85: Status 404 returned error can't find the container with id 84cf3efd6278b1466041acbe0f6cf3012c52e12f13616fc0edf24b59e8ba2c85 Oct 31 00:33:39.234485 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.234455 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:39.234535 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.234518 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:39.234570 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.234545 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:39.276515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.276492 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pjwds"] Oct 31 00:33:39.319546 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.319517 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pjwds" Oct 31 00:33:39.333034 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.332992 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pjwds"] Oct 31 00:33:39.338908 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.338884 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6b4pn" Oct 31 00:33:39.369999 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.369964 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-trzc2\"" Oct 31 00:33:39.395237 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.395182 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" event={"ID":"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699","Type":"ContainerStarted","Data":"c3924615ed1dd5539ffef6c828e30e8681f8efe41fa32986f6a123bd74287c68"} Oct 31 00:33:39.395414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.395310 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" event={"ID":"bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699","Type":"ContainerStarted","Data":"84cf3efd6278b1466041acbe0f6cf3012c52e12f13616fc0edf24b59e8ba2c85"} Oct 31 00:33:39.483157 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.483071 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzk2m\" (UniqueName: \"kubernetes.io/projected/cfdcb397-0d07-42b6-b3a8-a052612eaaa0-kube-api-access-dzk2m\") pod \"authorino-57f7f8659f-pjwds\" (UID: \"cfdcb397-0d07-42b6-b3a8-a052612eaaa0\") " pod="kuadrant-system/authorino-57f7f8659f-pjwds" Oct 31 00:33:39.584808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.584701 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk2m\" (UniqueName: \"kubernetes.io/projected/cfdcb397-0d07-42b6-b3a8-a052612eaaa0-kube-api-access-dzk2m\") pod \"authorino-57f7f8659f-pjwds\" (UID: \"cfdcb397-0d07-42b6-b3a8-a052612eaaa0\") " pod="kuadrant-system/authorino-57f7f8659f-pjwds" Oct 31 00:33:39.647685 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.647647 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzk2m\" (UniqueName: \"kubernetes.io/projected/cfdcb397-0d07-42b6-b3a8-a052612eaaa0-kube-api-access-dzk2m\") pod \"authorino-57f7f8659f-pjwds\" (UID: \"cfdcb397-0d07-42b6-b3a8-a052612eaaa0\") " pod="kuadrant-system/authorino-57f7f8659f-pjwds" Oct 31 00:33:39.650465 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.650438 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pjwds" Oct 31 00:33:39.663126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.663080 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" podStartSLOduration=1.663065416 podStartE2EDuration="1.663065416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:39.661814514 +0000 UTC m=+311.864064941" watchObservedRunningTime="2026-10-31 00:33:39.663065416 +0000 UTC m=+311.865315849" Oct 31 00:33:39.759091 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:39.759025 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:40.049136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:40.049113 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pjwds"] Oct 31 00:33:40.051331 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:40.051284 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfdcb397_0d07_42b6_b3a8_a052612eaaa0.slice/crio-917ae0f6242da1fc844a9a46f545832be525e99421444002b6376a3bd4f95473 WatchSource:0}: Error finding container 917ae0f6242da1fc844a9a46f545832be525e99421444002b6376a3bd4f95473: Status 404 returned error can't find the container with id 917ae0f6242da1fc844a9a46f545832be525e99421444002b6376a3bd4f95473 Oct 31 00:33:40.345988 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:40.345909 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6k4fb" Oct 31 00:33:40.401339 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:40.401301 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pjwds" event={"ID":"cfdcb397-0d07-42b6-b3a8-a052612eaaa0","Type":"ContainerStarted","Data":"917ae0f6242da1fc844a9a46f545832be525e99421444002b6376a3bd4f95473"} Oct 31 00:33:40.763532 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:40.763501 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:41.406130 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:41.406096 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:41.407041 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:41.407019 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fgwg4f" Oct 31 00:33:44.423653 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:44.423614 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pjwds" event={"ID":"cfdcb397-0d07-42b6-b3a8-a052612eaaa0","Type":"ContainerStarted","Data":"3a40c7d0e2f86968010904a4c82b268856e6d4141cd852a9b6146f3c7c11654c"} Oct 31 00:33:44.573519 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:44.573463 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-pjwds" podStartSLOduration=1.930127945 podStartE2EDuration="5.573447765s" totalImagesPullingTime="3.64331982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:44.572400284 +0000 UTC m=+316.774650716" watchObservedRunningTime="2026-10-31 00:33:44.573447765 +0000 UTC m=+316.775698197" Oct 31 00:33:45.371701 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:45.371670 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zp68g" Oct 31 00:33:47.192977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.192939 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pjwds"] Oct 31 00:33:47.193376 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.193144 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-pjwds" podUID="cfdcb397-0d07-42b6-b3a8-a052612eaaa0" containerName="authorino" containerID="cri-o://3a40c7d0e2f86968010904a4c82b268856e6d4141cd852a9b6146f3c7c11654c" gracePeriod=30 Oct 31 00:33:47.439184 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.439152 5890 generic.go:356] "Generic (PLEG): container finished" podID="cfdcb397-0d07-42b6-b3a8-a052612eaaa0" containerID="3a40c7d0e2f86968010904a4c82b268856e6d4141cd852a9b6146f3c7c11654c" exitCode=0 Oct 31 00:33:47.439317 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.439187 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pjwds" event={"ID":"cfdcb397-0d07-42b6-b3a8-a052612eaaa0","Type":"ContainerDied","Data":"3a40c7d0e2f86968010904a4c82b268856e6d4141cd852a9b6146f3c7c11654c"} Oct 31 00:33:47.439317 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.439233 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pjwds" event={"ID":"cfdcb397-0d07-42b6-b3a8-a052612eaaa0","Type":"ContainerDied","Data":"917ae0f6242da1fc844a9a46f545832be525e99421444002b6376a3bd4f95473"} Oct 31 00:33:47.439317 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.439253 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917ae0f6242da1fc844a9a46f545832be525e99421444002b6376a3bd4f95473" Oct 31 00:33:47.448087 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.448025 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pjwds" Oct 31 00:33:47.458066 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.458046 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfdcb397-0d07-42b6-b3a8-a052612eaaa0-kube-api-access-dzk2m\" (UniqueName: \"kubernetes.io/projected/cfdcb397-0d07-42b6-b3a8-a052612eaaa0-kube-api-access-dzk2m\") pod \"cfdcb397-0d07-42b6-b3a8-a052612eaaa0\" (UID: \"cfdcb397-0d07-42b6-b3a8-a052612eaaa0\") " Oct 31 00:33:47.460116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.460092 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdcb397-0d07-42b6-b3a8-a052612eaaa0-kube-api-access-dzk2m" pod "cfdcb397-0d07-42b6-b3a8-a052612eaaa0" (UID: "cfdcb397-0d07-42b6-b3a8-a052612eaaa0"). InnerVolumeSpecName "kube-api-access-dzk2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:47.558614 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:47.558582 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzk2m\" (UniqueName: \"kubernetes.io/projected/cfdcb397-0d07-42b6-b3a8-a052612eaaa0-kube-api-access-dzk2m\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:48.444434 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:48.444402 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pjwds" Oct 31 00:33:48.600315 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:48.600281 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pjwds"] Oct 31 00:33:48.648244 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:48.648211 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pjwds"] Oct 31 00:33:49.932425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:49.932387 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfdcb397-0d07-42b6-b3a8-a052612eaaa0" path="/var/lib/kubelet/pods/cfdcb397-0d07-42b6-b3a8-a052612eaaa0/volumes" Oct 31 00:33:55.789446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.789410 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct"] Oct 31 00:33:55.789862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.789811 5890 cpu_manager.go:414] "containerMap: removing container" podUID="cfdcb397-0d07-42b6-b3a8-a052612eaaa0" containerName="authorino" Oct 31 00:33:55.789862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.789827 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdcb397-0d07-42b6-b3a8-a052612eaaa0" containerName="authorino" Oct 31 00:33:55.789948 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.789940 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdcb397-0d07-42b6-b3a8-a052612eaaa0" containerName="authorino" Oct 31 00:33:55.796049 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.796029 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.820891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.820859 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4pc\" (UniqueName: \"kubernetes.io/projected/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-kube-api-access-zw4pc\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.821025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.820901 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-manager-config\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.821025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.820919 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-metrics-certs\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.821025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.820962 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-cert\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.848077 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.848047 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct"] Oct 31 00:33:55.894031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.894002 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:33:55.894170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.894050 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-9sjf9\"" Oct 31 00:33:55.894170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.894127 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 00:33:55.894286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.894187 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 00:33:55.894286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.894265 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 00:33:55.894358 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.894343 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 00:33:55.921444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.921418 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4pc\" (UniqueName: \"kubernetes.io/projected/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-kube-api-access-zw4pc\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.921543 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.921456 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-manager-config\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.921543 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.921482 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-metrics-certs\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.921543 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.921507 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-cert\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.922137 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.922115 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-manager-config\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.923983 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.923960 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-cert\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.924061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.924041 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-metrics-certs\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:55.982143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:55.982113 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4pc\" (UniqueName: \"kubernetes.io/projected/6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb-kube-api-access-zw4pc\") pod \"jobset-controller-manager-86d5fddf5b-6j2ct\" (UID: \"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb\") " pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:56.107129 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:56.107046 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:33:56.473902 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:56.473875 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct"] Oct 31 00:33:56.475779 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:33:56.475747 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd5900c_4b37_4f4c_9e2c_8fae4169d4bb.slice/crio-95e90ea1c0f118cb2dea9c3a6b6c4d1938706392b8f8a47bbb90a3fb0f2b9012 WatchSource:0}: Error finding container 95e90ea1c0f118cb2dea9c3a6b6c4d1938706392b8f8a47bbb90a3fb0f2b9012: Status 404 returned error can't find the container with id 95e90ea1c0f118cb2dea9c3a6b6c4d1938706392b8f8a47bbb90a3fb0f2b9012 Oct 31 00:33:56.482908 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:33:56.482884 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" event={"ID":"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb","Type":"ContainerStarted","Data":"95e90ea1c0f118cb2dea9c3a6b6c4d1938706392b8f8a47bbb90a3fb0f2b9012"} Oct 31 00:34:03.519642 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:34:03.519603 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" event={"ID":"6fd5900c-4b37-4f4c-9e2c-8fae4169d4bb","Type":"ContainerStarted","Data":"7fc8e5e1c7967ce48e4f7272a9aff35cf8f6a1e0d486e0d8299665d95451e5e9"} Oct 31 00:34:03.520151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:34:03.519743 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:34:03.633974 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:34:03.633922 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" podStartSLOduration=2.267690913 podStartE2EDuration="8.633907385s" totalImagesPullingTime="6.366216472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:03.632186105 +0000 UTC m=+335.834436538" watchObservedRunningTime="2026-10-31 00:34:03.633907385 +0000 UTC m=+335.836157815" Oct 31 00:34:07.016723 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:34:07.016683 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:34:14.529778 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:34:14.529737 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-86d5fddf5b-6j2ct" Oct 31 00:34:37.017167 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:34:37.017124 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:35:07.017479 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:35:07.017442 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:35:12.870421 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.870387 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h677g"] Oct 31 00:35:12.873909 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.873891 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:12.928515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.928482 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h677g"] Oct 31 00:35:12.966058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.966027 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dd7557-9f36-490f-8786-332d2a64de50-tls-certs\") pod \"model-serving-api-75bd89fdfc-h677g\" (UID: \"70dd7557-9f36-490f-8786-332d2a64de50\") " pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:12.966204 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.966186 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtzv\" (UniqueName: \"kubernetes.io/projected/70dd7557-9f36-490f-8786-332d2a64de50-kube-api-access-5wtzv\") pod \"model-serving-api-75bd89fdfc-h677g\" (UID: \"70dd7557-9f36-490f-8786-332d2a64de50\") " pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:12.974558 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.974525 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:35:12.974692 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.974648 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 00:35:12.974747 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.974711 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:35:12.974867 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:12.974847 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ds4rw\"" Oct 31 00:35:13.066914 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:13.066877 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dd7557-9f36-490f-8786-332d2a64de50-tls-certs\") pod \"model-serving-api-75bd89fdfc-h677g\" (UID: \"70dd7557-9f36-490f-8786-332d2a64de50\") " pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:13.067070 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:13.066947 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtzv\" (UniqueName: \"kubernetes.io/projected/70dd7557-9f36-490f-8786-332d2a64de50-kube-api-access-5wtzv\") pod \"model-serving-api-75bd89fdfc-h677g\" (UID: \"70dd7557-9f36-490f-8786-332d2a64de50\") " pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:13.069371 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:13.069342 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dd7557-9f36-490f-8786-332d2a64de50-tls-certs\") pod \"model-serving-api-75bd89fdfc-h677g\" (UID: \"70dd7557-9f36-490f-8786-332d2a64de50\") " pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:13.125849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:13.125757 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtzv\" (UniqueName: \"kubernetes.io/projected/70dd7557-9f36-490f-8786-332d2a64de50-kube-api-access-5wtzv\") pod \"model-serving-api-75bd89fdfc-h677g\" (UID: \"70dd7557-9f36-490f-8786-332d2a64de50\") " pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:13.189123 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:13.189086 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:13.554106 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:13.554076 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-h677g"] Oct 31 00:35:13.556884 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:35:13.556847 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dd7557_9f36_490f_8786_332d2a64de50.slice/crio-12bef78cbb7e6bb8c9e789c1f44b2b0efa83c5e6c90891ec04057e9ca063de9d WatchSource:0}: Error finding container 12bef78cbb7e6bb8c9e789c1f44b2b0efa83c5e6c90891ec04057e9ca063de9d: Status 404 returned error can't find the container with id 12bef78cbb7e6bb8c9e789c1f44b2b0efa83c5e6c90891ec04057e9ca063de9d Oct 31 00:35:13.866145 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:13.866065 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-h677g" event={"ID":"70dd7557-9f36-490f-8786-332d2a64de50","Type":"ContainerStarted","Data":"12bef78cbb7e6bb8c9e789c1f44b2b0efa83c5e6c90891ec04057e9ca063de9d"} Oct 31 00:35:16.887701 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:16.887609 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-h677g" event={"ID":"70dd7557-9f36-490f-8786-332d2a64de50","Type":"ContainerStarted","Data":"156902fcca06816f21c49d7e31591ff926c4770aff1d4115ad29a028d7955691"} Oct 31 00:35:16.888182 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:16.887718 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:16.999675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:16.999621 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-h677g" podStartSLOduration=2.029093761 podStartE2EDuration="4.999604854s" totalImagesPullingTime="2.970511093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:16.999526446 +0000 UTC m=+409.201776890" watchObservedRunningTime="2026-10-31 00:35:16.999604854 +0000 UTC m=+409.201855288" Oct 31 00:35:27.896860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:27.896829 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-h677g" Oct 31 00:35:36.535271 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.535232 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fjqs8"] Oct 31 00:35:36.539253 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.539233 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" Oct 31 00:35:36.591885 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.591852 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fjqs8"] Oct 31 00:35:36.637572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.637419 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-trzc2\"" Oct 31 00:35:36.702943 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.702907 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhnm\" (UniqueName: \"kubernetes.io/projected/57c55009-9b3c-40b5-9ace-2926061cb0fb-kube-api-access-2fhnm\") pod \"authorino-6f6944bcdd-fjqs8\" (UID: \"57c55009-9b3c-40b5-9ace-2926061cb0fb\") " pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" Oct 31 00:35:36.804280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.804192 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhnm\" (UniqueName: \"kubernetes.io/projected/57c55009-9b3c-40b5-9ace-2926061cb0fb-kube-api-access-2fhnm\") pod \"authorino-6f6944bcdd-fjqs8\" (UID: \"57c55009-9b3c-40b5-9ace-2926061cb0fb\") " pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" Oct 31 00:35:36.863233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.863193 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhnm\" (UniqueName: \"kubernetes.io/projected/57c55009-9b3c-40b5-9ace-2926061cb0fb-kube-api-access-2fhnm\") pod \"authorino-6f6944bcdd-fjqs8\" (UID: \"57c55009-9b3c-40b5-9ace-2926061cb0fb\") " pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" Oct 31 00:35:36.874636 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.874611 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" Oct 31 00:35:36.902459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.902415 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fjqs8"] Oct 31 00:35:36.945610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.945516 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c8d8fdbb7-65f6v"] Oct 31 00:35:36.949305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:36.949284 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" Oct 31 00:35:37.002570 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.002536 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c8d8fdbb7-65f6v"] Oct 31 00:35:37.020064 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:35:37.020029 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:35:37.106926 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.106848 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrmm\" (UniqueName: \"kubernetes.io/projected/eba49171-06e6-47cc-b553-d651206af750-kube-api-access-tbrmm\") pod \"authorino-c8d8fdbb7-65f6v\" (UID: \"eba49171-06e6-47cc-b553-d651206af750\") " pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" Oct 31 00:35:37.207856 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.207813 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrmm\" (UniqueName: \"kubernetes.io/projected/eba49171-06e6-47cc-b553-d651206af750-kube-api-access-tbrmm\") pod \"authorino-c8d8fdbb7-65f6v\" (UID: \"eba49171-06e6-47cc-b553-d651206af750\") " pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" Oct 31 00:35:37.239834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.239749 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fjqs8"] Oct 31 00:35:37.240056 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:35:37.240034 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57c55009_9b3c_40b5_9ace_2926061cb0fb.slice/crio-1b1c3147d77128a85584d4277a9b0f0de8f47e1a9eb1ebb68612bf572cb1f1a2 WatchSource:0}: Error finding container 1b1c3147d77128a85584d4277a9b0f0de8f47e1a9eb1ebb68612bf572cb1f1a2: Status 404 returned error can't find the container with id 1b1c3147d77128a85584d4277a9b0f0de8f47e1a9eb1ebb68612bf572cb1f1a2 Oct 31 00:35:37.267072 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.267040 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrmm\" (UniqueName: \"kubernetes.io/projected/eba49171-06e6-47cc-b553-d651206af750-kube-api-access-tbrmm\") pod \"authorino-c8d8fdbb7-65f6v\" (UID: \"eba49171-06e6-47cc-b553-d651206af750\") " pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" Oct 31 00:35:37.362636 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.362557 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c8d8fdbb7-65f6v"] Oct 31 00:35:37.362807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.362775 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" Oct 31 00:35:37.391010 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.390978 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cff9fb997-zjfl9"] Oct 31 00:35:37.394599 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.394580 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.446270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.446208 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 00:35:37.447938 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.447907 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cff9fb997-zjfl9"] Oct 31 00:35:37.510826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.510756 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86f32c97-5610-491e-9fdd-8dc6d88db23e-tls-cert\") pod \"authorino-5cff9fb997-zjfl9\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.511019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.510869 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srshr\" (UniqueName: \"kubernetes.io/projected/86f32c97-5610-491e-9fdd-8dc6d88db23e-kube-api-access-srshr\") pod \"authorino-5cff9fb997-zjfl9\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.611972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.611936 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86f32c97-5610-491e-9fdd-8dc6d88db23e-tls-cert\") pod \"authorino-5cff9fb997-zjfl9\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.611972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.611976 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srshr\" (UniqueName: \"kubernetes.io/projected/86f32c97-5610-491e-9fdd-8dc6d88db23e-kube-api-access-srshr\") pod \"authorino-5cff9fb997-zjfl9\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.614378 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.614318 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86f32c97-5610-491e-9fdd-8dc6d88db23e-tls-cert\") pod \"authorino-5cff9fb997-zjfl9\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.670865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.670829 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srshr\" (UniqueName: \"kubernetes.io/projected/86f32c97-5610-491e-9fdd-8dc6d88db23e-kube-api-access-srshr\") pod \"authorino-5cff9fb997-zjfl9\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.708435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.708403 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:35:37.731071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:37.731039 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c8d8fdbb7-65f6v"] Oct 31 00:35:37.731933 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:35:37.731906 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeba49171_06e6_47cc_b553_d651206af750.slice/crio-d1fdd3b16858da5add5163f2cc587e8303ba17bd86af0828342b880ce2d9e857 WatchSource:0}: Error finding container d1fdd3b16858da5add5163f2cc587e8303ba17bd86af0828342b880ce2d9e857: Status 404 returned error can't find the container with id d1fdd3b16858da5add5163f2cc587e8303ba17bd86af0828342b880ce2d9e857 Oct 31 00:35:38.016230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:38.016180 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" event={"ID":"eba49171-06e6-47cc-b553-d651206af750","Type":"ContainerStarted","Data":"d1fdd3b16858da5add5163f2cc587e8303ba17bd86af0828342b880ce2d9e857"} Oct 31 00:35:38.017451 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:38.017419 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" event={"ID":"57c55009-9b3c-40b5-9ace-2926061cb0fb","Type":"ContainerStarted","Data":"1b1c3147d77128a85584d4277a9b0f0de8f47e1a9eb1ebb68612bf572cb1f1a2"} Oct 31 00:35:38.080217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:38.080192 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cff9fb997-zjfl9"] Oct 31 00:35:38.099648 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:35:38.099613 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86f32c97_5610_491e_9fdd_8dc6d88db23e.slice/crio-a38952e14c2e5d5090214c0b0722c523c6ca974a050884f20b92ade974506f41 WatchSource:0}: Error finding container a38952e14c2e5d5090214c0b0722c523c6ca974a050884f20b92ade974506f41: Status 404 returned error can't find the container with id a38952e14c2e5d5090214c0b0722c523c6ca974a050884f20b92ade974506f41 Oct 31 00:35:39.023286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:39.023247 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" event={"ID":"86f32c97-5610-491e-9fdd-8dc6d88db23e","Type":"ContainerStarted","Data":"a38952e14c2e5d5090214c0b0722c523c6ca974a050884f20b92ade974506f41"} Oct 31 00:35:39.024862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:39.024838 5890 generic.go:356] "Generic (PLEG): container finished" podID="aedae545-0e22-4793-a720-230efee85446" containerID="7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff" exitCode=0 Oct 31 00:35:39.024983 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:39.024899 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" event={"ID":"aedae545-0e22-4793-a720-230efee85446","Type":"ContainerDied","Data":"7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff"} Oct 31 00:35:39.025232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:39.025219 5890 scope.go:122] "RemoveContainer" containerID="7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff" Oct 31 00:35:39.026232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:39.026196 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:35:39.026334 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:39.026287 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:35:39.026396 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:39.026339 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:35:40.033422 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.033331 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" event={"ID":"aedae545-0e22-4793-a720-230efee85446","Type":"ContainerStarted","Data":"9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c"} Oct 31 00:35:40.035080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.035045 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" event={"ID":"eba49171-06e6-47cc-b553-d651206af750","Type":"ContainerStarted","Data":"f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9"} Oct 31 00:35:40.035221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.035115 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" podUID="eba49171-06e6-47cc-b553-d651206af750" containerName="authorino" containerID="cri-o://f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9" gracePeriod=30 Oct 31 00:35:40.036860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.036824 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" event={"ID":"57c55009-9b3c-40b5-9ace-2926061cb0fb","Type":"ContainerStarted","Data":"be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db"} Oct 31 00:35:40.036860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.036825 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" podUID="57c55009-9b3c-40b5-9ace-2926061cb0fb" containerName="authorino" containerID="cri-o://be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db" gracePeriod=30 Oct 31 00:35:40.040524 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.038865 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" event={"ID":"86f32c97-5610-491e-9fdd-8dc6d88db23e","Type":"ContainerStarted","Data":"427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e"} Oct 31 00:35:40.383522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.383448 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" podStartSLOduration=2.54973157 podStartE2EDuration="4.383427126s" totalImagesPullingTime="1.833695556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:40.381968525 +0000 UTC m=+432.584218959" watchObservedRunningTime="2026-10-31 00:35:40.383427126 +0000 UTC m=+432.585677560" Oct 31 00:35:40.383645 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.383588 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" podStartSLOduration=2.721301627 podStartE2EDuration="4.383582879s" totalImagesPullingTime="1.662281252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:40.269493267 +0000 UTC m=+432.471743700" watchObservedRunningTime="2026-10-31 00:35:40.383582879 +0000 UTC m=+432.585833311" Oct 31 00:35:40.400722 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.400699 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" Oct 31 00:35:40.415401 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.415373 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" Oct 31 00:35:40.496619 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.496556 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" podStartSLOduration=2.002129311 podStartE2EDuration="3.4965406s" totalImagesPullingTime="1.494411289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:40.494879015 +0000 UTC m=+432.697129443" watchObservedRunningTime="2026-10-31 00:35:40.4965406 +0000 UTC m=+432.698791033" Oct 31 00:35:40.538576 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.538531 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c55009-9b3c-40b5-9ace-2926061cb0fb-kube-api-access-2fhnm\" (UniqueName: \"kubernetes.io/projected/57c55009-9b3c-40b5-9ace-2926061cb0fb-kube-api-access-2fhnm\") pod \"57c55009-9b3c-40b5-9ace-2926061cb0fb\" (UID: \"57c55009-9b3c-40b5-9ace-2926061cb0fb\") " Oct 31 00:35:40.538752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.538587 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba49171-06e6-47cc-b553-d651206af750-kube-api-access-tbrmm\" (UniqueName: \"kubernetes.io/projected/eba49171-06e6-47cc-b553-d651206af750-kube-api-access-tbrmm\") pod \"eba49171-06e6-47cc-b553-d651206af750\" (UID: \"eba49171-06e6-47cc-b553-d651206af750\") " Oct 31 00:35:40.540913 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.540878 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba49171-06e6-47cc-b553-d651206af750-kube-api-access-tbrmm" pod "eba49171-06e6-47cc-b553-d651206af750" (UID: "eba49171-06e6-47cc-b553-d651206af750"). InnerVolumeSpecName "kube-api-access-tbrmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:40.541159 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.541128 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c55009-9b3c-40b5-9ace-2926061cb0fb-kube-api-access-2fhnm" pod "57c55009-9b3c-40b5-9ace-2926061cb0fb" (UID: "57c55009-9b3c-40b5-9ace-2926061cb0fb"). InnerVolumeSpecName "kube-api-access-2fhnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:40.639727 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.639651 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhnm\" (UniqueName: \"kubernetes.io/projected/57c55009-9b3c-40b5-9ace-2926061cb0fb-kube-api-access-2fhnm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.639727 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.639684 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbrmm\" (UniqueName: \"kubernetes.io/projected/eba49171-06e6-47cc-b553-d651206af750-kube-api-access-tbrmm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.970957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.970909 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:35:40.972147 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.972117 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:40.972267 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:40.972172 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:41.069288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.069243 5890 generic.go:356] "Generic (PLEG): container finished" podID="eba49171-06e6-47cc-b553-d651206af750" containerID="f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9" exitCode=0 Oct 31 00:35:41.069773 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.069363 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" event={"ID":"eba49171-06e6-47cc-b553-d651206af750","Type":"ContainerDied","Data":"f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9"} Oct 31 00:35:41.069773 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.069371 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" Oct 31 00:35:41.069773 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.069411 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c8d8fdbb7-65f6v" event={"ID":"eba49171-06e6-47cc-b553-d651206af750","Type":"ContainerDied","Data":"d1fdd3b16858da5add5163f2cc587e8303ba17bd86af0828342b880ce2d9e857"} Oct 31 00:35:41.069773 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.069438 5890 scope.go:122] "RemoveContainer" containerID="f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9" Oct 31 00:35:41.071670 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.071644 5890 generic.go:356] "Generic (PLEG): container finished" podID="57c55009-9b3c-40b5-9ace-2926061cb0fb" containerID="be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db" exitCode=0 Oct 31 00:35:41.071818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.071748 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" Oct 31 00:35:41.071818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.071803 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" event={"ID":"57c55009-9b3c-40b5-9ace-2926061cb0fb","Type":"ContainerDied","Data":"be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db"} Oct 31 00:35:41.071946 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.071850 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fjqs8" event={"ID":"57c55009-9b3c-40b5-9ace-2926061cb0fb","Type":"ContainerDied","Data":"1b1c3147d77128a85584d4277a9b0f0de8f47e1a9eb1ebb68612bf572cb1f1a2"} Oct 31 00:35:41.080707 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.080455 5890 scope.go:122] "RemoveContainer" containerID="f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9" Oct 31 00:35:41.080798 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:35:41.080762 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9\": container with ID starting with f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9 not found: ID does not exist" containerID="f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9" Oct 31 00:35:41.080854 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.080809 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9"} err="failed to get container status \"f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9\": rpc error: code = NotFound desc = could not find container \"f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9\": container with ID starting with f3bb8396804bf21f5470c5cb6e269bb1a6870ad0dd6f7f380c5716a4380f0bf9 not found: ID does not exist" Oct 31 00:35:41.080854 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.080834 5890 scope.go:122] "RemoveContainer" containerID="be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db" Oct 31 00:35:41.088757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.088743 5890 scope.go:122] "RemoveContainer" containerID="be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db" Oct 31 00:35:41.089038 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:35:41.089020 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db\": container with ID starting with be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db not found: ID does not exist" containerID="be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db" Oct 31 00:35:41.089109 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.089046 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db"} err="failed to get container status \"be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db\": rpc error: code = NotFound desc = could not find container \"be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db\": container with ID starting with be4d7e4424db8f61f3a0b34cd0f8b63622eb88f7a339f6991414c717d87289db not found: ID does not exist" Oct 31 00:35:41.232214 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.232135 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c8d8fdbb7-65f6v"] Oct 31 00:35:41.236284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.236253 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c8d8fdbb7-65f6v"] Oct 31 00:35:41.340664 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.340629 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fjqs8"] Oct 31 00:35:41.342124 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.342104 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fjqs8"] Oct 31 00:35:41.932333 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.932301 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c55009-9b3c-40b5-9ace-2926061cb0fb" path="/var/lib/kubelet/pods/57c55009-9b3c-40b5-9ace-2926061cb0fb/volumes" Oct 31 00:35:41.932637 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.932625 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba49171-06e6-47cc-b553-d651206af750" path="/var/lib/kubelet/pods/eba49171-06e6-47cc-b553-d651206af750/volumes" Oct 31 00:35:41.971413 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.971377 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:41.971583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:41.971440 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:42.971433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:42.971385 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:42.971815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:42.971474 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:43.023883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:43.023851 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:35:43.971487 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:43.971442 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:43.971999 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:43.971504 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:44.971421 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:44.971389 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:35:44.971701 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:44.971450 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 31 00:35:45.975548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:45.975519 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:35:45.976466 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:35:45.976446 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 00:36:07.016243 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:36:07.016203 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:36:37.017890 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:36:37.017846 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:37:07.016475 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:37:07.016435 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:37:37.018091 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:37:37.018049 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:37:50.967536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.967491 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl"] Oct 31 00:37:50.968170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.967884 5890 cpu_manager.go:414] "containerMap: removing container" podUID="57c55009-9b3c-40b5-9ace-2926061cb0fb" containerName="authorino" Oct 31 00:37:50.968170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.967896 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c55009-9b3c-40b5-9ace-2926061cb0fb" containerName="authorino" Oct 31 00:37:50.968170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.967906 5890 cpu_manager.go:414] "containerMap: removing container" podUID="eba49171-06e6-47cc-b553-d651206af750" containerName="authorino" Oct 31 00:37:50.968170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.967911 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba49171-06e6-47cc-b553-d651206af750" containerName="authorino" Oct 31 00:37:50.968170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.968014 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c55009-9b3c-40b5-9ace-2926061cb0fb" containerName="authorino" Oct 31 00:37:50.968170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.968024 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba49171-06e6-47cc-b553-d651206af750" containerName="authorino" Oct 31 00:37:50.971031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:50.971008 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.021625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.021592 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 00:37:51.021822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.021599 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 00:37:51.062274 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.062226 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdz69\" (UniqueName: \"kubernetes.io/projected/b7848d63-5555-4ae9-8ba1-934e520c43f7-kube-api-access-vdz69\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.062440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.062289 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.062440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.062377 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.062440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.062418 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7848d63-5555-4ae9-8ba1-934e520c43f7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.062549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.062515 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.062582 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.062543 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.069394 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.069366 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 00:37:51.069535 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.069491 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pzv4k\"" Oct 31 00:37:51.071119 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.071097 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl"] Oct 31 00:37:51.163609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.163575 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.163609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.163608 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.163911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.163667 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdz69\" (UniqueName: \"kubernetes.io/projected/b7848d63-5555-4ae9-8ba1-934e520c43f7-kube-api-access-vdz69\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.163911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.163701 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.163911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.163720 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.163911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.163742 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7848d63-5555-4ae9-8ba1-934e520c43f7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.164126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.164091 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.164171 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.164139 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.164492 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.164469 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.166013 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.165994 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7848d63-5555-4ae9-8ba1-934e520c43f7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.166273 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.166253 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7848d63-5555-4ae9-8ba1-934e520c43f7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.225048 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.224969 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdz69\" (UniqueName: \"kubernetes.io/projected/b7848d63-5555-4ae9-8ba1-934e520c43f7-kube-api-access-vdz69\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qpkjl\" (UID: \"b7848d63-5555-4ae9-8ba1-934e520c43f7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.284329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.284293 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:37:51.661483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.661381 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl"] Oct 31 00:37:51.667913 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:37:51.667879 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7848d63_5555_4ae9_8ba1_934e520c43f7.slice/crio-68d501e5ce3b42691604a9b1e7c42bdc1c1c8e5d50c132508d0e0fc32f2055f0 WatchSource:0}: Error finding container 68d501e5ce3b42691604a9b1e7c42bdc1c1c8e5d50c132508d0e0fc32f2055f0: Status 404 returned error can't find the container with id 68d501e5ce3b42691604a9b1e7c42bdc1c1c8e5d50c132508d0e0fc32f2055f0 Oct 31 00:37:51.729006 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:51.728969 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" event={"ID":"b7848d63-5555-4ae9-8ba1-934e520c43f7","Type":"ContainerStarted","Data":"68d501e5ce3b42691604a9b1e7c42bdc1c1c8e5d50c132508d0e0fc32f2055f0"} Oct 31 00:37:53.746358 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:53.746318 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" event={"ID":"b7848d63-5555-4ae9-8ba1-934e520c43f7","Type":"ContainerStarted","Data":"b4ddf4d01a8052755774b3704480846b9cab628e935d510fe5c29cfa449baf5c"} Oct 31 00:37:53.859589 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:53.859537 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" podStartSLOduration=2.306683526 podStartE2EDuration="3.859524705s" totalImagesPullingTime="1.552841179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:53.85916298 +0000 UTC m=+566.061413414" watchObservedRunningTime="2026-10-31 00:37:53.859524705 +0000 UTC m=+566.061775138" Oct 31 00:37:54.285369 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:37:54.285331 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:38:01.284744 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.284702 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:38:01.297186 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.297161 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:38:01.672467 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.672438 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mfrb7"] Oct 31 00:38:01.676197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.676178 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.726409 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.726378 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 00:38:01.769761 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.769722 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vj6w\" (UniqueName: \"kubernetes.io/projected/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-kube-api-access-4vj6w\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.769941 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.769776 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-home\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.769941 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.769905 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.770028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.769948 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.770065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.770025 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.770109 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.770063 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.775492 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.775459 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mfrb7"] Oct 31 00:38:01.802618 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.802584 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qpkjl" Oct 31 00:38:01.870741 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.870703 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.870919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.870749 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.870919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.870801 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.870919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.870841 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.871079 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.870929 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vj6w\" (UniqueName: \"kubernetes.io/projected/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-kube-api-access-4vj6w\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.871079 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.871013 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-home\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.871184 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.871162 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.871295 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.871269 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.871353 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.871316 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-home\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.873148 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.873128 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.873210 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.873196 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.958918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.958837 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vj6w\" (UniqueName: \"kubernetes.io/projected/ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4-kube-api-access-4vj6w\") pod \"echo-simulator-kserve-64b6f8f96c-mfrb7\" (UID: \"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:01.988989 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:01.988956 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:02.361756 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:02.361728 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mfrb7"] Oct 31 00:38:02.363779 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:38:02.363749 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdbfdaf_a6ca_4997_9cfd_acc37e1915b4.slice/crio-986c6c0d244dc79eae84b2cf505e4989ae7d0da2d23f4b6e4a0dad475de33999 WatchSource:0}: Error finding container 986c6c0d244dc79eae84b2cf505e4989ae7d0da2d23f4b6e4a0dad475de33999: Status 404 returned error can't find the container with id 986c6c0d244dc79eae84b2cf505e4989ae7d0da2d23f4b6e4a0dad475de33999 Oct 31 00:38:02.798123 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:02.798087 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" event={"ID":"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4","Type":"ContainerStarted","Data":"986c6c0d244dc79eae84b2cf505e4989ae7d0da2d23f4b6e4a0dad475de33999"} Oct 31 00:38:04.813073 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:04.812972 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" event={"ID":"ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4","Type":"ContainerStarted","Data":"b795b8405bc74a2d124d56921e187ae4e2e181fdd82a9130b0101388491d785f"} Oct 31 00:38:04.927746 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:04.927689 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" podStartSLOduration=1.740635997 podStartE2EDuration="3.927675603s" totalImagesPullingTime="2.187039606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:04.926440905 +0000 UTC m=+577.128691353" watchObservedRunningTime="2026-10-31 00:38:04.927675603 +0000 UTC m=+577.129926035" Oct 31 00:38:04.989236 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:04.989203 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:07.017342 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:38:07.017291 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:38:08.301123 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.301092 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2"] Oct 31 00:38:08.304751 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.304731 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.403170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.403142 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 00:38:08.405096 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.405074 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2"] Oct 31 00:38:08.434071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.434038 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7b6\" (UniqueName: \"kubernetes.io/projected/b661f84e-8c0e-484f-a531-afc81020e2f5-kube-api-access-fx7b6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.434071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.434071 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b661f84e-8c0e-484f-a531-afc81020e2f5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.434293 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.434109 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.434293 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.434263 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.434387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.434324 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.434442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.434420 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535351 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535302 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535376 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535454 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535485 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7b6\" (UniqueName: \"kubernetes.io/projected/b661f84e-8c0e-484f-a531-afc81020e2f5-kube-api-access-fx7b6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535525 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b661f84e-8c0e-484f-a531-afc81020e2f5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535774 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535584 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535854 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535825 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535909 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535873 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.535951 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.535919 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.537832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.537807 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b661f84e-8c0e-484f-a531-afc81020e2f5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.538135 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.538111 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b661f84e-8c0e-484f-a531-afc81020e2f5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.594714 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.594658 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7b6\" (UniqueName: \"kubernetes.io/projected/b661f84e-8c0e-484f-a531-afc81020e2f5-kube-api-access-fx7b6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2\" (UID: \"b661f84e-8c0e-484f-a531-afc81020e2f5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.616637 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.616604 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:08.988519 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:08.988490 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2"] Oct 31 00:38:08.990202 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:38:08.990173 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb661f84e_8c0e_484f_a531_afc81020e2f5.slice/crio-d5bc651c0aa239b8a269f4b3bb0a51abd949b17508346b8ec65cba414a6e77e3 WatchSource:0}: Error finding container d5bc651c0aa239b8a269f4b3bb0a51abd949b17508346b8ec65cba414a6e77e3: Status 404 returned error can't find the container with id d5bc651c0aa239b8a269f4b3bb0a51abd949b17508346b8ec65cba414a6e77e3 Oct 31 00:38:09.840507 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:09.840414 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" event={"ID":"b661f84e-8c0e-484f-a531-afc81020e2f5","Type":"ContainerStarted","Data":"30452b63b0b823a5884df95bc50c1e65f51319e15c0e709ee53f80f59d5c505b"} Oct 31 00:38:09.840507 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:09.840461 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" event={"ID":"b661f84e-8c0e-484f-a531-afc81020e2f5","Type":"ContainerStarted","Data":"d5bc651c0aa239b8a269f4b3bb0a51abd949b17508346b8ec65cba414a6e77e3"} Oct 31 00:38:09.954606 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:09.954550 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podStartSLOduration=1.358120229 podStartE2EDuration="1.954538207s" totalImagesPullingTime="596.417978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:09.953928279 +0000 UTC m=+582.156178712" watchObservedRunningTime="2026-10-31 00:38:09.954538207 +0000 UTC m=+582.156788685" Oct 31 00:38:10.617719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:10.617685 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:11.989910 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:11.989876 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:12.005529 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:12.005501 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:12.866278 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:12.866252 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mfrb7" Oct 31 00:38:18.616951 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:18.616909 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:18.630737 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:18.630711 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:18.902353 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:18.902275 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:38:27.999084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:27.999057 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:38:27.999084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:27.999080 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:38:34.358518 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.358476 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f7f7dc8cd-g96sp"] Oct 31 00:38:34.365143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.365116 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.415809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.415762 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7f7dc8cd-g96sp"] Oct 31 00:38:34.481027 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.480982 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95fk\" (UniqueName: \"kubernetes.io/projected/91127689-97c8-4ca7-82a1-6d2d22f3a9c4-kube-api-access-s95fk\") pod \"authorino-5f7f7dc8cd-g96sp\" (UID: \"91127689-97c8-4ca7-82a1-6d2d22f3a9c4\") " pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.481185 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.481034 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91127689-97c8-4ca7-82a1-6d2d22f3a9c4-tls-cert\") pod \"authorino-5f7f7dc8cd-g96sp\" (UID: \"91127689-97c8-4ca7-82a1-6d2d22f3a9c4\") " pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.582390 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.582355 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91127689-97c8-4ca7-82a1-6d2d22f3a9c4-tls-cert\") pod \"authorino-5f7f7dc8cd-g96sp\" (UID: \"91127689-97c8-4ca7-82a1-6d2d22f3a9c4\") " pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.582574 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.582510 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95fk\" (UniqueName: \"kubernetes.io/projected/91127689-97c8-4ca7-82a1-6d2d22f3a9c4-kube-api-access-s95fk\") pod \"authorino-5f7f7dc8cd-g96sp\" (UID: \"91127689-97c8-4ca7-82a1-6d2d22f3a9c4\") " pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.584916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.584890 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91127689-97c8-4ca7-82a1-6d2d22f3a9c4-tls-cert\") pod \"authorino-5f7f7dc8cd-g96sp\" (UID: \"91127689-97c8-4ca7-82a1-6d2d22f3a9c4\") " pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.627128 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.627065 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k"] Oct 31 00:38:34.631234 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.631213 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.641113 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.641081 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95fk\" (UniqueName: \"kubernetes.io/projected/91127689-97c8-4ca7-82a1-6d2d22f3a9c4-kube-api-access-s95fk\") pod \"authorino-5f7f7dc8cd-g96sp\" (UID: \"91127689-97c8-4ca7-82a1-6d2d22f3a9c4\") " pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.677952 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.677926 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" Oct 31 00:38:34.709653 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.709618 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k"] Oct 31 00:38:34.784438 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784405 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4e48e6-2013-426f-b584-10c4f04e0034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784457 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784535 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784554 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784573 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784606 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784766 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784933 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784829 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmkm\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-kube-api-access-kpmkm\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.784933 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.784864 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886242 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886287 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886311 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmkm\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-kube-api-access-kpmkm\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886328 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886355 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4e48e6-2013-426f-b584-10c4f04e0034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886398 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886433 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886463 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886492 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.886905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.886756 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.887044 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.887013 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.887158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.887044 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.887158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.887084 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.887158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.887142 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4e48e6-2013-426f-b584-10c4f04e0034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.888730 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.888709 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.889021 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.889003 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.945193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.945160 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:34.945327 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:34.945297 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmkm\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-kube-api-access-kpmkm\") pod \"maas-default-gateway-openshift-default-6cc458d887-b687k\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:35.046158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.046133 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7f7dc8cd-g96sp"] Oct 31 00:38:35.048615 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:38:35.048586 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91127689_97c8_4ca7_82a1_6d2d22f3a9c4.slice/crio-f9b8f105d6817d907d3ae280a2fa63e7a656960c9846c8640fcf5e64ee7a4d57 WatchSource:0}: Error finding container f9b8f105d6817d907d3ae280a2fa63e7a656960c9846c8640fcf5e64ee7a4d57: Status 404 returned error can't find the container with id f9b8f105d6817d907d3ae280a2fa63e7a656960c9846c8640fcf5e64ee7a4d57 Oct 31 00:38:35.049906 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.049889 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:38:35.244302 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.244265 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:35.664564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.664540 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k"] Oct 31 00:38:35.667323 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:38:35.667291 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4e48e6_2013_426f_b584_10c4f04e0034.slice/crio-0e9c2b76985f493837e345d3bc3a54cd18e3c2273b9a5504bd14306c0d698176 WatchSource:0}: Error finding container 0e9c2b76985f493837e345d3bc3a54cd18e3c2273b9a5504bd14306c0d698176: Status 404 returned error can't find the container with id 0e9c2b76985f493837e345d3bc3a54cd18e3c2273b9a5504bd14306c0d698176 Oct 31 00:38:35.669996 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.669964 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:35.670079 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.670032 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:35.670079 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.670063 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:35.982369 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.982325 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" event={"ID":"3d4e48e6-2013-426f-b584-10c4f04e0034","Type":"ContainerStarted","Data":"b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87"} Oct 31 00:38:35.982529 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.982404 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" event={"ID":"3d4e48e6-2013-426f-b584-10c4f04e0034","Type":"ContainerStarted","Data":"0e9c2b76985f493837e345d3bc3a54cd18e3c2273b9a5504bd14306c0d698176"} Oct 31 00:38:35.983596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:35.983572 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" event={"ID":"91127689-97c8-4ca7-82a1-6d2d22f3a9c4","Type":"ContainerStarted","Data":"f9b8f105d6817d907d3ae280a2fa63e7a656960c9846c8640fcf5e64ee7a4d57"} Oct 31 00:38:36.101163 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:36.101108 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podStartSLOduration=2.101092694 podStartE2EDuration="2.101092694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:36.100295824 +0000 UTC m=+608.302546256" watchObservedRunningTime="2026-10-31 00:38:36.101092694 +0000 UTC m=+608.303343126" Oct 31 00:38:36.264133 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:36.264035 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:36.266320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:36.266290 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:36.266537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:36.266512 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:36.991144 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:36.991091 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" event={"ID":"91127689-97c8-4ca7-82a1-6d2d22f3a9c4","Type":"ContainerStarted","Data":"ac8740a9dfae95ed846e3750a133ea363d747b49db88ec21b5118d144194fe9a"} Oct 31 00:38:37.018125 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:38:37.018086 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:38:37.103535 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.103425 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f7f7dc8cd-g96sp" podStartSLOduration=1.369037952 podStartE2EDuration="3.103411415s" totalImagesPullingTime="1.734373463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:37.101728745 +0000 UTC m=+609.303979179" watchObservedRunningTime="2026-10-31 00:38:37.103411415 +0000 UTC m=+609.305661896" Oct 31 00:38:37.132409 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.132375 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cff9fb997-zjfl9"] Oct 31 00:38:37.132575 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.132556 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" podUID="86f32c97-5610-491e-9fdd-8dc6d88db23e" containerName="authorino" containerID="cri-o://427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e" gracePeriod=30 Oct 31 00:38:37.266332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.265872 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:37.266332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.265943 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:37.430141 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.430117 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:38:37.508436 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.508394 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f32c97-5610-491e-9fdd-8dc6d88db23e-tls-cert\" (UniqueName: \"kubernetes.io/secret/86f32c97-5610-491e-9fdd-8dc6d88db23e-tls-cert\") pod \"86f32c97-5610-491e-9fdd-8dc6d88db23e\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " Oct 31 00:38:37.508626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.508495 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f32c97-5610-491e-9fdd-8dc6d88db23e-kube-api-access-srshr\" (UniqueName: \"kubernetes.io/projected/86f32c97-5610-491e-9fdd-8dc6d88db23e-kube-api-access-srshr\") pod \"86f32c97-5610-491e-9fdd-8dc6d88db23e\" (UID: \"86f32c97-5610-491e-9fdd-8dc6d88db23e\") " Oct 31 00:38:37.510936 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.510901 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f32c97-5610-491e-9fdd-8dc6d88db23e-kube-api-access-srshr" pod "86f32c97-5610-491e-9fdd-8dc6d88db23e" (UID: "86f32c97-5610-491e-9fdd-8dc6d88db23e"). InnerVolumeSpecName "kube-api-access-srshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:37.520671 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.520647 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f32c97-5610-491e-9fdd-8dc6d88db23e-tls-cert" pod "86f32c97-5610-491e-9fdd-8dc6d88db23e" (UID: "86f32c97-5610-491e-9fdd-8dc6d88db23e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:37.609966 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.609929 5890 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86f32c97-5610-491e-9fdd-8dc6d88db23e-tls-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:37.609966 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.609963 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srshr\" (UniqueName: \"kubernetes.io/projected/86f32c97-5610-491e-9fdd-8dc6d88db23e-kube-api-access-srshr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:37.998438 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.998399 5890 generic.go:356] "Generic (PLEG): container finished" podID="86f32c97-5610-491e-9fdd-8dc6d88db23e" containerID="427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e" exitCode=0 Oct 31 00:38:37.998976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.998453 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" Oct 31 00:38:37.998976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.998486 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" event={"ID":"86f32c97-5610-491e-9fdd-8dc6d88db23e","Type":"ContainerDied","Data":"427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e"} Oct 31 00:38:37.998976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.998546 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cff9fb997-zjfl9" event={"ID":"86f32c97-5610-491e-9fdd-8dc6d88db23e","Type":"ContainerDied","Data":"a38952e14c2e5d5090214c0b0722c523c6ca974a050884f20b92ade974506f41"} Oct 31 00:38:37.998976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:37.998579 5890 scope.go:122] "RemoveContainer" containerID="427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e" Oct 31 00:38:38.008739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:38.008717 5890 scope.go:122] "RemoveContainer" containerID="427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e" Oct 31 00:38:38.009149 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:38:38.009122 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e\": container with ID starting with 427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e not found: ID does not exist" containerID="427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e" Oct 31 00:38:38.009237 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:38.009161 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e"} err="failed to get container status \"427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e\": rpc error: code = NotFound desc = could not find container \"427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e\": container with ID starting with 427459fbd49009ac3ac998155ef3e55da294602e49b2096917801c76cb8a7a2e not found: ID does not exist" Oct 31 00:38:38.176186 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:38.176139 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cff9fb997-zjfl9"] Oct 31 00:38:38.177893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:38.177857 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cff9fb997-zjfl9"] Oct 31 00:38:38.248222 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:38.248186 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:38.248490 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:38.248461 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:39.254211 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:39.254173 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:39.254869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:39.254833 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:39.932615 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:39.932580 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f32c97-5610-491e-9fdd-8dc6d88db23e" path="/var/lib/kubelet/pods/86f32c97-5610-491e-9fdd-8dc6d88db23e/volumes" Oct 31 00:38:40.249414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:40.249310 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:40.249696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:40.249648 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:41.264809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:41.260210 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:41.264809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:41.260274 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:42.245295 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:42.245249 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:42.245495 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:42.245317 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:43.244745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:43.244698 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 00:38:43.245258 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:43.244814 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 00:38:44.248906 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:44.248876 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:44.249363 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:44.249176 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:38:44.250047 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:38:44.250023 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 00:39:07.017739 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:39:07.017698 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:39:14.736019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.735928 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z"] Oct 31 00:39:14.736587 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.736564 5890 cpu_manager.go:414] "containerMap: removing container" podUID="86f32c97-5610-491e-9fdd-8dc6d88db23e" containerName="authorino" Oct 31 00:39:14.736634 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.736592 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f32c97-5610-491e-9fdd-8dc6d88db23e" containerName="authorino" Oct 31 00:39:14.736742 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.736728 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f32c97-5610-491e-9fdd-8dc6d88db23e" containerName="authorino" Oct 31 00:39:14.739769 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.739750 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785206 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785169 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-envoy\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785352 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785222 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-podinfo\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785352 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785250 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59cw\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-kube-api-access-v59cw\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785437 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785343 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-certs\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785437 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785378 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-data\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785436 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-token\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785474 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-socket\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785518 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0dc4d09-1427-4a19-9903-597de4b83e45-istiod-ca-cert\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.785664 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.785638 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-credential-socket\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.789864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.789843 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-24d78330-openshift-default-dockercfg-pk9pn\"" Oct 31 00:39:14.840153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.840118 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z"] Oct 31 00:39:14.886513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886482 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-envoy\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.886700 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886529 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-podinfo\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.886700 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886660 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59cw\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-kube-api-access-v59cw\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.886845 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886715 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-certs\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.886845 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886746 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-data\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.886845 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886805 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-token\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.887018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886863 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-socket\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.887018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886940 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0dc4d09-1427-4a19-9903-597de4b83e45-istiod-ca-cert\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.887018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.886994 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-credential-socket\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.887179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.887126 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-certs\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.887398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.887374 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-data\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.887541 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.887520 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-socket\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.888303 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.888281 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0dc4d09-1427-4a19-9903-597de4b83e45-istiod-ca-cert\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.888980 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.888959 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-envoy\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.889063 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.888982 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-credential-socket\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.889341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.889321 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-podinfo\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.945674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.945640 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-token\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:14.945862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:14.945753 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59cw\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-kube-api-access-v59cw\") pod \"e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:15.053617 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.053526 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:15.532586 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.532562 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z"] Oct 31 00:39:15.534372 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:39:15.534342 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dc4d09_1427_4a19_9903_597de4b83e45.slice/crio-a3dcb61fd9e18746ddad391a90d79ae687a9d2bb88c66d46c42da0507dd64574 WatchSource:0}: Error finding container a3dcb61fd9e18746ddad391a90d79ae687a9d2bb88c66d46c42da0507dd64574: Status 404 returned error can't find the container with id a3dcb61fd9e18746ddad391a90d79ae687a9d2bb88c66d46c42da0507dd64574 Oct 31 00:39:15.536565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.536528 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:15.536668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.536611 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:15.536668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.536658 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:15.668601 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.668564 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv"] Oct 31 00:39:15.672917 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.672892 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.722825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.722796 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-30d0c8aa-gw-openshift-default-dockercfg-jwrcb\"" Oct 31 00:39:15.728329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728297 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-socket\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.728905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728348 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-credential-socket\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.728905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728570 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-token\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.728905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728613 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-envoy\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.728905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728682 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff42s\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-kube-api-access-ff42s\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.728905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728741 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-certs\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.728905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728764 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-podinfo\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.728905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.728902 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562ef5c-d803-4a72-9597-76ebcdaff508-istiod-ca-cert\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.729636 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.729614 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-data\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.730494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.730475 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv"] Oct 31 00:39:15.831193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831100 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-token\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831145 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-envoy\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831171 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff42s\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-kube-api-access-ff42s\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831206 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-certs\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831231 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-podinfo\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831283 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562ef5c-d803-4a72-9597-76ebcdaff508-istiod-ca-cert\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831335 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-data\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831399 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-socket\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.831826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831432 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-credential-socket\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.832155 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831825 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-credential-socket\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.832155 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831865 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-data\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.832155 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.831967 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-socket\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.832155 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.832115 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-certs\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.832541 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.832519 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562ef5c-d803-4a72-9597-76ebcdaff508-istiod-ca-cert\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.833796 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.833760 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-envoy\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.834122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.834090 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-podinfo\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.890103 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.890066 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-token\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.890291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.890147 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff42s\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-kube-api-access-ff42s\") pod \"e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:15.986992 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:15.986952 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:16.207621 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:16.207585 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" event={"ID":"e0dc4d09-1427-4a19-9903-597de4b83e45","Type":"ContainerStarted","Data":"8b9b9c872e44cd46cfe1eda417751d832093d5ddea65532da7e533e79fc9e1b6"} Oct 31 00:39:16.207803 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:16.207642 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" event={"ID":"e0dc4d09-1427-4a19-9903-597de4b83e45","Type":"ContainerStarted","Data":"a3dcb61fd9e18746ddad391a90d79ae687a9d2bb88c66d46c42da0507dd64574"} Oct 31 00:39:16.324350 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:16.324297 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" podStartSLOduration=2.32428147 podStartE2EDuration="2.32428147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:16.323186376 +0000 UTC m=+648.525436808" watchObservedRunningTime="2026-10-31 00:39:16.32428147 +0000 UTC m=+648.526532316" Oct 31 00:39:16.456885 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:16.456860 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv"] Oct 31 00:39:16.458543 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:39:16.458472 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0562ef5c_d803_4a72_9597_76ebcdaff508.slice/crio-595290fb0eede028fa7fed514873c826d0026e1f5d0da978086d41fdfe512909 WatchSource:0}: Error finding container 595290fb0eede028fa7fed514873c826d0026e1f5d0da978086d41fdfe512909: Status 404 returned error can't find the container with id 595290fb0eede028fa7fed514873c826d0026e1f5d0da978086d41fdfe512909 Oct 31 00:39:16.468241 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:16.468196 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:16.468371 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:16.468291 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:16.468371 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:16.468327 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:17.054420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.054389 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:17.059290 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.059262 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:17.216236 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.216199 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" event={"ID":"0562ef5c-d803-4a72-9597-76ebcdaff508","Type":"ContainerStarted","Data":"926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07"} Oct 31 00:39:17.216434 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.216267 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" event={"ID":"0562ef5c-d803-4a72-9597-76ebcdaff508","Type":"ContainerStarted","Data":"595290fb0eede028fa7fed514873c826d0026e1f5d0da978086d41fdfe512909"} Oct 31 00:39:17.216635 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.216610 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:17.217507 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.217487 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:39:17.332956 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.332854 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podStartSLOduration=2.33284159 podStartE2EDuration="2.33284159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:17.33193929 +0000 UTC m=+649.534189757" watchObservedRunningTime="2026-10-31 00:39:17.33284159 +0000 UTC m=+649.535092023" Oct 31 00:39:17.987886 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.987851 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:17.993367 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:17.993340 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:18.221818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:18.221756 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:18.223173 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:18.223136 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:22.384865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.384744 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6"] Oct 31 00:39:22.389305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.389282 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.485592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.485555 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6"] Oct 31 00:39:22.500184 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500150 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd118214-d05a-4dc1-9177-7d998352f52f-istiod-ca-cert\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500355 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500190 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-istio-token\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500355 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500322 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-data\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500495 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500378 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-certs\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500495 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500409 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvpn\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-kube-api-access-znvpn\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500495 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500442 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd118214-d05a-4dc1-9177-7d998352f52f-istio-podinfo\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500522 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-socket\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500550 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-envoy\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.500668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.500663 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-credential-socket\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.530236 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.530205 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2b9202f4-openshift-default-dockercfg-x46jd\"" Oct 31 00:39:22.601701 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.601669 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-istio-token\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.601916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.601729 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-data\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.601916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.601774 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-certs\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.601916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.601845 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvpn\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-kube-api-access-znvpn\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.601916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.601891 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd118214-d05a-4dc1-9177-7d998352f52f-istio-podinfo\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.601916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.601921 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-socket\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.601916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.601957 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-envoy\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.602367 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.602032 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-credential-socket\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.602367 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.602089 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd118214-d05a-4dc1-9177-7d998352f52f-istiod-ca-cert\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.602367 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.602235 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-data\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.602564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.602535 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-certs\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.602831 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.602780 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd118214-d05a-4dc1-9177-7d998352f52f-istiod-ca-cert\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.603142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.603106 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-credential-socket\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.603236 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.603123 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-socket\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.605021 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.604993 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-envoy\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.605125 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.605085 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd118214-d05a-4dc1-9177-7d998352f52f-istio-podinfo\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.660324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.660245 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-istio-token\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.663429 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.663403 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvpn\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-kube-api-access-znvpn\") pod \"e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:22.703264 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:22.703237 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:23.137756 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.137705 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6"] Oct 31 00:39:23.141060 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:39:23.141025 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd118214_d05a_4dc1_9177_7d998352f52f.slice/crio-dd4d8ffdcb9fe06d7653d44af16ce82bc2f3b7262bf434334ef8550d683ebd51 WatchSource:0}: Error finding container dd4d8ffdcb9fe06d7653d44af16ce82bc2f3b7262bf434334ef8550d683ebd51: Status 404 returned error can't find the container with id dd4d8ffdcb9fe06d7653d44af16ce82bc2f3b7262bf434334ef8550d683ebd51 Oct 31 00:39:23.143311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.143274 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:23.143401 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.143352 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:23.143401 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.143386 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:23.252590 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.252555 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" event={"ID":"fd118214-d05a-4dc1-9177-7d998352f52f","Type":"ContainerStarted","Data":"d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954"} Oct 31 00:39:23.252767 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.252611 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" event={"ID":"fd118214-d05a-4dc1-9177-7d998352f52f","Type":"ContainerStarted","Data":"dd4d8ffdcb9fe06d7653d44af16ce82bc2f3b7262bf434334ef8550d683ebd51"} Oct 31 00:39:23.378267 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.378211 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podStartSLOduration=1.3781975 podStartE2EDuration="1.3781975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:23.375213188 +0000 UTC m=+655.577463620" watchObservedRunningTime="2026-10-31 00:39:23.3781975 +0000 UTC m=+655.580447933" Oct 31 00:39:23.704050 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:23.704009 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:24.709362 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:24.709328 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:25.264009 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:25.263969 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:25.265020 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:25.264996 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:39:27.621477 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.621442 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9"] Oct 31 00:39:27.625203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.625187 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:27.725692 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.725658 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-24d78330\"" Oct 31 00:39:27.725898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.725816 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-24d78330-dockercfg-wrs4h\"" Oct 31 00:39:27.727913 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.727877 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9"] Oct 31 00:39:27.765807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.761450 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxhdb\" (UniqueName: \"kubernetes.io/projected/f04f3d86-2cda-4423-99ee-b26d45d25c70-kube-api-access-pxhdb\") pod \"payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:27.765807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.761615 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04f3d86-2cda-4423-99ee-b26d45d25c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:27.862954 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.862920 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxhdb\" (UniqueName: \"kubernetes.io/projected/f04f3d86-2cda-4423-99ee-b26d45d25c70-kube-api-access-pxhdb\") pod \"payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:27.863139 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.863062 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04f3d86-2cda-4423-99ee-b26d45d25c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:27.864031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.864003 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04f3d86-2cda-4423-99ee-b26d45d25c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:27.922448 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.922418 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxhdb\" (UniqueName: \"kubernetes.io/projected/f04f3d86-2cda-4423-99ee-b26d45d25c70-kube-api-access-pxhdb\") pod \"payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:27.987981 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.987946 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-24d78330-dockercfg-wrs4h\"" Oct 31 00:39:27.990470 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:27.990442 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:28.361674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:28.361648 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9"] Oct 31 00:39:28.364116 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:39:28.364078 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04f3d86_2cda_4423_99ee_b26d45d25c70.slice/crio-e2bc9b33720bc7a7e6a83fff98686595c12e378dcdbc3ef21d879e1a9255b7b0 WatchSource:0}: Error finding container e2bc9b33720bc7a7e6a83fff98686595c12e378dcdbc3ef21d879e1a9255b7b0: Status 404 returned error can't find the container with id e2bc9b33720bc7a7e6a83fff98686595c12e378dcdbc3ef21d879e1a9255b7b0 Oct 31 00:39:29.285857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:29.285817 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" event={"ID":"f04f3d86-2cda-4423-99ee-b26d45d25c70","Type":"ContainerStarted","Data":"e2bc9b33720bc7a7e6a83fff98686595c12e378dcdbc3ef21d879e1a9255b7b0"} Oct 31 00:39:30.026212 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.026171 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h"] Oct 31 00:39:30.030493 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.030467 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.117835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.117117 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdv7\" (UniqueName: \"kubernetes.io/projected/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-kube-api-access-zsdv7\") pod \"payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.117835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.117201 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.163173 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.163116 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d6a47bf7\"" Oct 31 00:39:30.163173 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.163116 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d6a47bf7-dockercfg-hqrzh\"" Oct 31 00:39:30.166137 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.166110 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h"] Oct 31 00:39:30.180256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.180226 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg"] Oct 31 00:39:30.184111 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.184090 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.218563 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.217762 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.218563 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.218150 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdv7\" (UniqueName: \"kubernetes.io/projected/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-kube-api-access-zsdv7\") pod \"payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.218741 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.218622 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.242799 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.242231 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg"] Oct 31 00:39:30.282197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.278062 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdv7\" (UniqueName: \"kubernetes.io/projected/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-kube-api-access-zsdv7\") pod \"payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.319521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.319483 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9scm\" (UniqueName: \"kubernetes.io/projected/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-kube-api-access-v9scm\") pod \"payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.320006 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.319698 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-plugins-config-volume\") pod \"payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.348352 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.348289 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:30.421217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.420906 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-plugins-config-volume\") pod \"payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.421217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.420997 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9scm\" (UniqueName: \"kubernetes.io/projected/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-kube-api-access-v9scm\") pod \"payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.423843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.422505 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-plugins-config-volume\") pod \"payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.480636 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.480567 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9scm\" (UniqueName: \"kubernetes.io/projected/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-kube-api-access-v9scm\") pod \"payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.499234 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.499192 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:30.740675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.740648 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h"] Oct 31 00:39:30.742920 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:39:30.742887 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3758e5c3_3e1b_4917_b2d3_af19804e1cc1.slice/crio-6c8c2d2628695f667548d3a33fc60d73a848c41027cffb043a9b3408151fe391 WatchSource:0}: Error finding container 6c8c2d2628695f667548d3a33fc60d73a848c41027cffb043a9b3408151fe391: Status 404 returned error can't find the container with id 6c8c2d2628695f667548d3a33fc60d73a848c41027cffb043a9b3408151fe391 Oct 31 00:39:30.887309 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:30.887275 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg"] Oct 31 00:39:30.889218 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:39:30.889176 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1de83c2_0950_46d3_872b_6b7bf06d7fd1.slice/crio-a876e874b0159b4e9925e2644c5c2a11d417ed899fd312c4bc77bee7bc8cce97 WatchSource:0}: Error finding container a876e874b0159b4e9925e2644c5c2a11d417ed899fd312c4bc77bee7bc8cce97: Status 404 returned error can't find the container with id a876e874b0159b4e9925e2644c5c2a11d417ed899fd312c4bc77bee7bc8cce97 Oct 31 00:39:31.301987 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:31.301953 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" event={"ID":"b1de83c2-0950-46d3-872b-6b7bf06d7fd1","Type":"ContainerStarted","Data":"a876e874b0159b4e9925e2644c5c2a11d417ed899fd312c4bc77bee7bc8cce97"} Oct 31 00:39:31.303465 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:31.303429 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" event={"ID":"3758e5c3-3e1b-4917-b2d3-af19804e1cc1","Type":"ContainerStarted","Data":"6c8c2d2628695f667548d3a33fc60d73a848c41027cffb043a9b3408151fe391"} Oct 31 00:39:32.317329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.317289 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" event={"ID":"b1de83c2-0950-46d3-872b-6b7bf06d7fd1","Type":"ContainerStarted","Data":"2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336"} Oct 31 00:39:32.317836 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.317410 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:32.319376 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.319347 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" event={"ID":"f04f3d86-2cda-4423-99ee-b26d45d25c70","Type":"ContainerStarted","Data":"6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed"} Oct 31 00:39:32.319496 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.319473 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:32.320894 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.320869 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" event={"ID":"3758e5c3-3e1b-4917-b2d3-af19804e1cc1","Type":"ContainerStarted","Data":"d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf"} Oct 31 00:39:32.321007 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.321000 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:32.432280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.432227 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" podStartSLOduration=1.5625451780000001 podStartE2EDuration="2.432214495s" totalImagesPullingTime="869.669317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:32.430674905 +0000 UTC m=+664.632925352" watchObservedRunningTime="2026-10-31 00:39:32.432214495 +0000 UTC m=+664.634464928" Oct 31 00:39:32.543474 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.543406 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" podStartSLOduration=2.561201143 podStartE2EDuration="3.543389732s" totalImagesPullingTime="982.188589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:32.543198465 +0000 UTC m=+664.745448898" watchObservedRunningTime="2026-10-31 00:39:32.543389732 +0000 UTC m=+664.745640166" Oct 31 00:39:32.661862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:32.661694 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" podStartSLOduration=2.378474151 podStartE2EDuration="5.661674153s" totalImagesPullingTime="3.283200002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:32.660082434 +0000 UTC m=+664.862332868" watchObservedRunningTime="2026-10-31 00:39:32.661674153 +0000 UTC m=+664.863924586" Oct 31 00:39:37.018531 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:39:37.018494 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:39:40.384719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.384680 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q"] Oct 31 00:39:40.389376 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.389352 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.427806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.427240 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-metrics-certs\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.427806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.427295 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.427806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.427338 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176542b6-7815-4e3d-94c7-8fa987949922-openshift-service-ca\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.427806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.427487 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlz6\" (UniqueName: \"kubernetes.io/projected/176542b6-7815-4e3d-94c7-8fa987949922-kube-api-access-5dlz6\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.470806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.470531 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q"] Oct 31 00:39:40.497273 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.497240 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:39:40.497571 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.497552 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-24d78330-serving-cert\"" Oct 31 00:39:40.498371 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.498346 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f72zx\"" Oct 31 00:39:40.518529 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.518502 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:39:40.518668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.518619 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:39:40.528109 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.528073 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-metrics-certs\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.528238 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.528116 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.528238 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.528163 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176542b6-7815-4e3d-94c7-8fa987949922-openshift-service-ca\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.528329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.528319 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlz6\" (UniqueName: \"kubernetes.io/projected/176542b6-7815-4e3d-94c7-8fa987949922-kube-api-access-5dlz6\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.528398 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:39:40.528317 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-serving-cert: secret "maas-api-e2e-ait-24d78330-serving-cert" not found Oct 31 00:39:40.528497 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:39:40.528468 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls podName:176542b6-7815-4e3d-94c7-8fa987949922 nodeName:}" failed. No retries permitted until 2026-10-31 00:39:41.028444528 +0000 UTC m=+673.230694961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls") pod "maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" (UID: "176542b6-7815-4e3d-94c7-8fa987949922") : secret "maas-api-e2e-ait-24d78330-serving-cert" not found Oct 31 00:39:40.529043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.529021 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176542b6-7815-4e3d-94c7-8fa987949922-openshift-service-ca\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.530658 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.530630 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-metrics-certs\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:40.614286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:40.599610 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlz6\" (UniqueName: \"kubernetes.io/projected/176542b6-7815-4e3d-94c7-8fa987949922-kube-api-access-5dlz6\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:41.035158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:41.035119 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:41.037557 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:41.037529 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls\") pod \"maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:41.306672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:41.306596 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:41.683816 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:41.682681 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q"] Oct 31 00:39:41.690542 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:39:41.690502 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176542b6_7815_4e3d_94c7_8fa987949922.slice/crio-09978401e13b156f2476f772feb90efd85e74f8e5051d3731ff4bd3301e32fe6 WatchSource:0}: Error finding container 09978401e13b156f2476f772feb90efd85e74f8e5051d3731ff4bd3301e32fe6: Status 404 returned error can't find the container with id 09978401e13b156f2476f772feb90efd85e74f8e5051d3731ff4bd3301e32fe6 Oct 31 00:39:42.383894 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:42.383840 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" event={"ID":"176542b6-7815-4e3d-94c7-8fa987949922","Type":"ContainerStarted","Data":"09978401e13b156f2476f772feb90efd85e74f8e5051d3731ff4bd3301e32fe6"} Oct 31 00:39:43.331764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:43.331727 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:39:43.332254 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:43.331868 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:39:43.332254 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:43.332026 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:39:46.411034 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:46.410985 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" event={"ID":"176542b6-7815-4e3d-94c7-8fa987949922","Type":"ContainerStarted","Data":"5e53001d22efd895bb0a3012383b7abd2e31eff5d12232d1b681b64339a08f1c"} Oct 31 00:39:46.411474 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:46.411060 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:46.531110 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:46.531047 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" podStartSLOduration=2.301009894 podStartE2EDuration="6.531027432s" totalImagesPullingTime="4.230017538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:46.529927653 +0000 UTC m=+678.732178087" watchObservedRunningTime="2026-10-31 00:39:46.531027432 +0000 UTC m=+678.733277867" Oct 31 00:39:49.262959 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:49.262918 5890 patch_prober.go:29] interesting pod/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:39:49.263431 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:49.262992 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:39:50.438753 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:50.438720 5890 generic.go:356] "Generic (PLEG): container finished" podID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerID="926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07" exitCode=0 Oct 31 00:39:50.439207 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:50.438816 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" event={"ID":"0562ef5c-d803-4a72-9597-76ebcdaff508","Type":"ContainerDied","Data":"926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07"} Oct 31 00:39:50.439261 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:50.439248 5890 scope.go:122] "RemoveContainer" containerID="926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07" Oct 31 00:39:50.440902 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:50.440867 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:50.440982 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:50.440965 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:50.441038 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:50.441022 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:51.459374 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:51.458166 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" event={"ID":"0562ef5c-d803-4a72-9597-76ebcdaff508","Type":"ContainerStarted","Data":"bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046"} Oct 31 00:39:51.992180 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:51.992144 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:51.998937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:51.998902 5890 patch_prober.go:29] interesting pod/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:39:51.999208 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:51.999179 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 00:39:52.432590 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:52.432556 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:39:53.004813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:52.990742 5890 patch_prober.go:29] interesting pod/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:39:53.004813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:52.990836 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 00:39:53.988821 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:53.988763 5890 patch_prober.go:29] interesting pod/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:39:53.989104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:53.989060 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 00:39:54.991231 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:54.991185 5890 patch_prober.go:29] interesting pod/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:39:54.991848 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:54.991810 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 00:39:56.000811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:55.991278 5890 patch_prober.go:29] interesting pod/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:39:56.000811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:55.991360 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 00:39:56.997074 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:56.997034 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:56.997628 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:56.997606 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:39:56.998292 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:39:56.998276 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:40:01.646494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.646398 5890 generic.go:356] "Generic (PLEG): container finished" podID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerID="8b9b9c872e44cd46cfe1eda417751d832093d5ddea65532da7e533e79fc9e1b6" exitCode=0 Oct 31 00:40:01.646494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.646473 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" event={"ID":"e0dc4d09-1427-4a19-9903-597de4b83e45","Type":"ContainerDied","Data":"8b9b9c872e44cd46cfe1eda417751d832093d5ddea65532da7e533e79fc9e1b6"} Oct 31 00:40:01.647286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.646908 5890 scope.go:122] "RemoveContainer" containerID="8b9b9c872e44cd46cfe1eda417751d832093d5ddea65532da7e533e79fc9e1b6" Oct 31 00:40:01.648344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.648305 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:01.648447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.648380 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:01.648447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.648422 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:01.648547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.648439 5890 generic.go:356] "Generic (PLEG): container finished" podID="fd118214-d05a-4dc1-9177-7d998352f52f" containerID="d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954" exitCode=0 Oct 31 00:40:01.648547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.648510 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" event={"ID":"fd118214-d05a-4dc1-9177-7d998352f52f","Type":"ContainerDied","Data":"d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954"} Oct 31 00:40:01.648932 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.648862 5890 scope.go:122] "RemoveContainer" containerID="d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954" Oct 31 00:40:01.650007 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.649980 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:01.650075 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.650046 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:01.650110 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:01.650088 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:02.217194 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:02.217146 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:40:02.702807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:02.702329 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" event={"ID":"fd118214-d05a-4dc1-9177-7d998352f52f","Type":"ContainerStarted","Data":"fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794"} Oct 31 00:40:02.703825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:02.703593 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:40:02.703825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:02.703697 5890 patch_prober.go:29] interesting pod/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:40:02.703825 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:02.703763 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 00:40:02.704930 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:02.704905 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" event={"ID":"e0dc4d09-1427-4a19-9903-597de4b83e45","Type":"ContainerStarted","Data":"a0cc58e3f00f19a2565c4432bb99b65df63f4eb4c2fcfae121de0708560a1173"} Oct 31 00:40:03.073868 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:03.067179 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:40:03.077809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:03.076913 5890 patch_prober.go:29] interesting pod/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:40:03.077809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:03.076985 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 00:40:03.704818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:03.704748 5890 patch_prober.go:29] interesting pod/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:40:03.705300 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:03.705269 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 00:40:04.065865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:04.065192 5890 patch_prober.go:29] interesting pod/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:40:04.065865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:04.065255 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 00:40:04.704420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:04.704378 5890 patch_prober.go:29] interesting pod/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:40:04.704617 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:04.704441 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 00:40:05.059866 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:05.057169 5890 patch_prober.go:29] interesting pod/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:40:05.059866 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:05.057236 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 00:40:05.704166 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:05.704130 5890 patch_prober.go:29] interesting pod/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:40:05.704354 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:05.704184 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 00:40:06.055292 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:06.055196 5890 patch_prober.go:29] interesting pod/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:40:06.055560 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:06.055528 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 00:40:06.705231 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:06.705179 5890 patch_prober.go:29] interesting pod/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:40:06.705735 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:06.705247 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 00:40:07.025183 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:07.025099 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:40:07.062811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:07.061214 5890 patch_prober.go:29] interesting pod/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:40:07.062811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:07.061275 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 00:40:07.704219 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:07.704180 5890 patch_prober.go:29] interesting pod/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:40:07.704444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:07.704240 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 00:40:08.058765 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:08.058690 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:40:08.059157 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:08.059031 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:40:08.059705 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:08.059686 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:40:08.708776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:08.708742 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:40:08.709188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:08.709168 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:40:08.709884 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:08.709865 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:40:10.565195 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.565156 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4"] Oct 31 00:40:10.568183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.568160 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.631440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.631405 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4"] Oct 31 00:40:10.668286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.668255 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:10.668286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.668272 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:40:10.674161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.674125 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.674287 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.674205 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.674287 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.674277 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndvh\" (UniqueName: \"kubernetes.io/projected/90d86a92-b73d-46ce-9537-1cf7da92821f-kube-api-access-zndvh\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.775096 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.775061 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.775305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.775186 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndvh\" (UniqueName: \"kubernetes.io/projected/90d86a92-b73d-46ce-9537-1cf7da92821f-kube-api-access-zndvh\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.775305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.775245 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.775829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.775774 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.777542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.777517 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.833996 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.833922 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndvh\" (UniqueName: \"kubernetes.io/projected/90d86a92-b73d-46ce-9537-1cf7da92821f-kube-api-access-zndvh\") pod \"maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:10.882476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:10.882450 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:11.257068 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:11.257044 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4"] Oct 31 00:40:11.259257 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:11.259222 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d86a92_b73d_46ce_9537_1cf7da92821f.slice/crio-cf437ad4e5ef372d34f41d6b5ce2bbd686e8cea46401d5ad65f5f7545bb21aaa WatchSource:0}: Error finding container cf437ad4e5ef372d34f41d6b5ce2bbd686e8cea46401d5ad65f5f7545bb21aaa: Status 404 returned error can't find the container with id cf437ad4e5ef372d34f41d6b5ce2bbd686e8cea46401d5ad65f5f7545bb21aaa Oct 31 00:40:11.757990 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:11.757954 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" event={"ID":"90d86a92-b73d-46ce-9537-1cf7da92821f","Type":"ContainerStarted","Data":"cf437ad4e5ef372d34f41d6b5ce2bbd686e8cea46401d5ad65f5f7545bb21aaa"} Oct 31 00:40:17.381289 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:17.381255 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg"] Oct 31 00:40:17.481606 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:17.481571 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h"] Oct 31 00:40:20.813297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:20.813216 5890 generic.go:356] "Generic (PLEG): container finished" podID="90d86a92-b73d-46ce-9537-1cf7da92821f" containerID="65e46998b186e904051db49809ae5288e4e96470c5a4dd4be91850855da5233b" exitCode=0 Oct 31 00:40:20.813297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:20.813262 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" event={"ID":"90d86a92-b73d-46ce-9537-1cf7da92821f","Type":"ContainerDied","Data":"65e46998b186e904051db49809ae5288e4e96470c5a4dd4be91850855da5233b"} Oct 31 00:40:21.958919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:21.958894 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:22.110282 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.110204 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-openshift-ca\") pod \"90d86a92-b73d-46ce-9537-1cf7da92821f\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " Oct 31 00:40:22.110454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.110319 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d86a92-b73d-46ce-9537-1cf7da92821f-kube-api-access-zndvh\" (UniqueName: \"kubernetes.io/projected/90d86a92-b73d-46ce-9537-1cf7da92821f-kube-api-access-zndvh\") pod \"90d86a92-b73d-46ce-9537-1cf7da92821f\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " Oct 31 00:40:22.110541 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.110452 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-xks-ca\") pod \"90d86a92-b73d-46ce-9537-1cf7da92821f\" (UID: \"90d86a92-b73d-46ce-9537-1cf7da92821f\") " Oct 31 00:40:22.110638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.110615 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-openshift-ca" pod "90d86a92-b73d-46ce-9537-1cf7da92821f" (UID: "90d86a92-b73d-46ce-9537-1cf7da92821f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:22.110773 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.110759 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-openshift-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.112526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.112500 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d86a92-b73d-46ce-9537-1cf7da92821f-kube-api-access-zndvh" pod "90d86a92-b73d-46ce-9537-1cf7da92821f" (UID: "90d86a92-b73d-46ce-9537-1cf7da92821f"). InnerVolumeSpecName "kube-api-access-zndvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:22.112593 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.112545 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-xks-ca" pod "90d86a92-b73d-46ce-9537-1cf7da92821f" (UID: "90d86a92-b73d-46ce-9537-1cf7da92821f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:22.212166 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.212129 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndvh\" (UniqueName: \"kubernetes.io/projected/90d86a92-b73d-46ce-9537-1cf7da92821f-kube-api-access-zndvh\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.212166 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.212163 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d86a92-b73d-46ce-9537-1cf7da92821f-maas-api-xks-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.381977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.381890 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" podUID="b1de83c2-0950-46d3-872b-6b7bf06d7fd1" containerName="payload-processing" containerID="cri-o://2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336" gracePeriod=25 Oct 31 00:40:22.482230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.482167 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" podUID="3758e5c3-3e1b-4917-b2d3-af19804e1cc1" containerName="payload-pre-processing" containerID="cri-o://d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf" gracePeriod=25 Oct 31 00:40:22.625104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.625079 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:40:22.717760 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.717724 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-kube-api-access-v9scm\" (UniqueName: \"kubernetes.io/projected/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-kube-api-access-v9scm\") pod \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " Oct 31 00:40:22.717950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.717776 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-plugins-config-volume\") pod \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\" (UID: \"b1de83c2-0950-46d3-872b-6b7bf06d7fd1\") " Oct 31 00:40:22.718199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.718175 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-plugins-config-volume" pod "b1de83c2-0950-46d3-872b-6b7bf06d7fd1" (UID: "b1de83c2-0950-46d3-872b-6b7bf06d7fd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:22.720383 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.720358 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-kube-api-access-v9scm" pod "b1de83c2-0950-46d3-872b-6b7bf06d7fd1" (UID: "b1de83c2-0950-46d3-872b-6b7bf06d7fd1"). InnerVolumeSpecName "kube-api-access-v9scm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:22.745955 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.745929 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:40:22.819487 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.819456 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-plugins-config-volume\") pod \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " Oct 31 00:40:22.819669 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.819561 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-kube-api-access-zsdv7\" (UniqueName: \"kubernetes.io/projected/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-kube-api-access-zsdv7\") pod \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\" (UID: \"3758e5c3-3e1b-4917-b2d3-af19804e1cc1\") " Oct 31 00:40:22.819803 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.819775 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9scm\" (UniqueName: \"kubernetes.io/projected/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-kube-api-access-v9scm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.819855 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.819817 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1de83c2-0950-46d3-872b-6b7bf06d7fd1-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.819941 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.819914 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-plugins-config-volume" pod "3758e5c3-3e1b-4917-b2d3-af19804e1cc1" (UID: "3758e5c3-3e1b-4917-b2d3-af19804e1cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:22.821626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.821603 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-kube-api-access-zsdv7" pod "3758e5c3-3e1b-4917-b2d3-af19804e1cc1" (UID: "3758e5c3-3e1b-4917-b2d3-af19804e1cc1"). InnerVolumeSpecName "kube-api-access-zsdv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:22.825721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.825697 5890 generic.go:356] "Generic (PLEG): container finished" podID="3758e5c3-3e1b-4917-b2d3-af19804e1cc1" containerID="d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf" exitCode=0 Oct 31 00:40:22.825850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.825751 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" Oct 31 00:40:22.825850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.825779 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" event={"ID":"3758e5c3-3e1b-4917-b2d3-af19804e1cc1","Type":"ContainerDied","Data":"d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf"} Oct 31 00:40:22.825956 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.825859 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h" event={"ID":"3758e5c3-3e1b-4917-b2d3-af19804e1cc1","Type":"ContainerDied","Data":"6c8c2d2628695f667548d3a33fc60d73a848c41027cffb043a9b3408151fe391"} Oct 31 00:40:22.825956 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.825887 5890 scope.go:122] "RemoveContainer" containerID="d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf" Oct 31 00:40:22.827240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.827224 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" Oct 31 00:40:22.827312 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.827271 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4" event={"ID":"90d86a92-b73d-46ce-9537-1cf7da92821f","Type":"ContainerDied","Data":"cf437ad4e5ef372d34f41d6b5ce2bbd686e8cea46401d5ad65f5f7545bb21aaa"} Oct 31 00:40:22.827622 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.827311 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf437ad4e5ef372d34f41d6b5ce2bbd686e8cea46401d5ad65f5f7545bb21aaa" Oct 31 00:40:22.829001 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.828971 5890 generic.go:356] "Generic (PLEG): container finished" podID="b1de83c2-0950-46d3-872b-6b7bf06d7fd1" containerID="2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336" exitCode=0 Oct 31 00:40:22.829115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.829011 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" event={"ID":"b1de83c2-0950-46d3-872b-6b7bf06d7fd1","Type":"ContainerDied","Data":"2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336"} Oct 31 00:40:22.829115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.829035 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" Oct 31 00:40:22.829115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.829064 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg" event={"ID":"b1de83c2-0950-46d3-872b-6b7bf06d7fd1","Type":"ContainerDied","Data":"a876e874b0159b4e9925e2644c5c2a11d417ed899fd312c4bc77bee7bc8cce97"} Oct 31 00:40:22.836607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.836591 5890 scope.go:122] "RemoveContainer" containerID="d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf" Oct 31 00:40:22.836955 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:22.836934 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf\": container with ID starting with d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf not found: ID does not exist" containerID="d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf" Oct 31 00:40:22.837010 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.836964 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf"} err="failed to get container status \"d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf\": rpc error: code = NotFound desc = could not find container \"d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf\": container with ID starting with d67cc59415d3659e9bb40f2fcdb79b764e7fb6fb34df25ab6c5eeb17b8e95acf not found: ID does not exist" Oct 31 00:40:22.837010 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.836981 5890 scope.go:122] "RemoveContainer" containerID="2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336" Oct 31 00:40:22.845749 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.845728 5890 scope.go:122] "RemoveContainer" containerID="2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336" Oct 31 00:40:22.846031 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:22.846012 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336\": container with ID starting with 2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336 not found: ID does not exist" containerID="2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336" Oct 31 00:40:22.846102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.846037 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336"} err="failed to get container status \"2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336\": rpc error: code = NotFound desc = could not find container \"2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336\": container with ID starting with 2acdd3bd057e03ded8f35f98a43193c40113b63f2702a709e72214edd8207336 not found: ID does not exist" Oct 31 00:40:22.921269 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.921192 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.921269 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.921221 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdv7\" (UniqueName: \"kubernetes.io/projected/3758e5c3-3e1b-4917-b2d3-af19804e1cc1-kube-api-access-zsdv7\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.965376 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.965348 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg"] Oct 31 00:40:22.969541 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:22.969514 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6a47bf7-7db7cc6c4f-jxxtg"] Oct 31 00:40:23.124124 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:23.124084 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h"] Oct 31 00:40:23.128452 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:23.128419 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6a47bf7-7dc674bd6c-4jj6h"] Oct 31 00:40:23.931293 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:23.931259 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3758e5c3-3e1b-4917-b2d3-af19804e1cc1" path="/var/lib/kubelet/pods/3758e5c3-3e1b-4917-b2d3-af19804e1cc1/volumes" Oct 31 00:40:23.931665 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:23.931649 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1de83c2-0950-46d3-872b-6b7bf06d7fd1" path="/var/lib/kubelet/pods/b1de83c2-0950-46d3-872b-6b7bf06d7fd1/volumes" Oct 31 00:40:28.006875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.006847 5890 scope.go:122] "RemoveContainer" containerID="3a40c7d0e2f86968010904a4c82b268856e6d4141cd852a9b6146f3c7c11654c" Oct 31 00:40:28.243926 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.243897 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz"] Oct 31 00:40:28.245589 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245524 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b1de83c2-0950-46d3-872b-6b7bf06d7fd1" containerName="payload-processing" Oct 31 00:40:28.245589 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245558 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1de83c2-0950-46d3-872b-6b7bf06d7fd1" containerName="payload-processing" Oct 31 00:40:28.245589 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245593 5890 cpu_manager.go:414] "containerMap: removing container" podUID="90d86a92-b73d-46ce-9537-1cf7da92821f" containerName="revoke-keys" Oct 31 00:40:28.245927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245602 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d86a92-b73d-46ce-9537-1cf7da92821f" containerName="revoke-keys" Oct 31 00:40:28.245927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245640 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3758e5c3-3e1b-4917-b2d3-af19804e1cc1" containerName="payload-pre-processing" Oct 31 00:40:28.245927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245648 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3758e5c3-3e1b-4917-b2d3-af19804e1cc1" containerName="payload-pre-processing" Oct 31 00:40:28.245927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245894 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1de83c2-0950-46d3-872b-6b7bf06d7fd1" containerName="payload-processing" Oct 31 00:40:28.245927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245916 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d86a92-b73d-46ce-9537-1cf7da92821f" containerName="revoke-keys" Oct 31 00:40:28.246243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.245935 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3758e5c3-3e1b-4917-b2d3-af19804e1cc1" containerName="payload-pre-processing" Oct 31 00:40:28.250600 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.250572 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.302153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.302122 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz"] Oct 31 00:40:28.346931 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.346903 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:28.347088 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.346958 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:40:28.376588 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.376558 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2st\" (UniqueName: \"kubernetes.io/projected/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-kube-api-access-wh2st\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.376744 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.376611 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.376744 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.376729 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.478077 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.478044 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2st\" (UniqueName: \"kubernetes.io/projected/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-kube-api-access-wh2st\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.478230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.478100 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.478230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.478135 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.478795 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.478760 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.480467 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.480445 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.536522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.536454 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2st\" (UniqueName: \"kubernetes.io/projected/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-kube-api-access-wh2st\") pod \"maas-api-revoke-keys-e2e-ait-95fa4445-pblpz\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.567043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.567016 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:28.935819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:28.935775 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz"] Oct 31 00:40:28.937310 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:28.937276 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c39ff9_35bb_452d_94b3_b41fb9e16b70.slice/crio-062732ff6c16710686cce27b408339e804b3d2869eeccf348028be12f7f4884c WatchSource:0}: Error finding container 062732ff6c16710686cce27b408339e804b3d2869eeccf348028be12f7f4884c: Status 404 returned error can't find the container with id 062732ff6c16710686cce27b408339e804b3d2869eeccf348028be12f7f4884c Oct 31 00:40:29.579689 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.579597 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv"] Oct 31 00:40:29.583122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.583097 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.641960 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.641919 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv"] Oct 31 00:40:29.681389 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.681359 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-29ece5-openshift-default-dockercfg-6vh8l\"" Oct 31 00:40:29.689804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.689767 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.689924 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.689816 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.689924 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.689903 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.690028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.690017 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.690065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.690044 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.690117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.690066 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.690117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.690088 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw78f\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-kube-api-access-nw78f\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.690235 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.690208 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.690277 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.690266 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.791614 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791572 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.791840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791660 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.791840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791695 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.791840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791723 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.791840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791762 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw78f\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-kube-api-access-nw78f\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791850 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791911 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.791995 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.792134 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.792173 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.792188 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792546 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.792525 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792687 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.792663 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.792854 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.792726 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.794358 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.794335 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.794949 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.794926 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.856447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.856369 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.857693 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.857668 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw78f\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-kube-api-access-nw78f\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:29.877564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.877533 5890 generic.go:356] "Generic (PLEG): container finished" podID="c5c39ff9-35bb-452d-94b3-b41fb9e16b70" containerID="48f68046442dd9da0498437bc1ea97365ee53931faa1a3992e3f6cb32c5a4ab7" exitCode=0 Oct 31 00:40:29.877713 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.877614 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" event={"ID":"c5c39ff9-35bb-452d-94b3-b41fb9e16b70","Type":"ContainerDied","Data":"48f68046442dd9da0498437bc1ea97365ee53931faa1a3992e3f6cb32c5a4ab7"} Oct 31 00:40:29.877713 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.877690 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" event={"ID":"c5c39ff9-35bb-452d-94b3-b41fb9e16b70","Type":"ContainerStarted","Data":"062732ff6c16710686cce27b408339e804b3d2869eeccf348028be12f7f4884c"} Oct 31 00:40:29.895166 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:29.895123 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:30.311350 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:30.311322 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv"] Oct 31 00:40:30.312905 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:30.312873 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6e2532_2e9b_468b_8c2b_9a0fbafe66c7.slice/crio-2e3d2431df6d326483bff329527280f6bfae038c32d0030254d6641bca145fad WatchSource:0}: Error finding container 2e3d2431df6d326483bff329527280f6bfae038c32d0030254d6641bca145fad: Status 404 returned error can't find the container with id 2e3d2431df6d326483bff329527280f6bfae038c32d0030254d6641bca145fad Oct 31 00:40:30.315197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:30.315160 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:30.315280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:30.315234 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:30.315280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:30.315263 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:30.884764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:30.884726 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" event={"ID":"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7","Type":"ContainerStarted","Data":"0ebd8a6ccebe6eb6443fe36868eceec141883f3c653882c9aab600050d0d4015"} Oct 31 00:40:30.885187 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:30.884824 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" event={"ID":"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7","Type":"ContainerStarted","Data":"2e3d2431df6d326483bff329527280f6bfae038c32d0030254d6641bca145fad"} Oct 31 00:40:30.895875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:30.895848 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:31.013000 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.012951 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podStartSLOduration=2.012937885 podStartE2EDuration="2.012937885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:31.011493688 +0000 UTC m=+723.213744131" watchObservedRunningTime="2026-10-31 00:40:31.012937885 +0000 UTC m=+723.215188308" Oct 31 00:40:31.023506 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.023485 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:31.206942 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.206908 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-xks-ca\") pod \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " Oct 31 00:40:31.207115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.206958 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-openshift-ca\") pod \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " Oct 31 00:40:31.207115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.207013 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-kube-api-access-wh2st\" (UniqueName: \"kubernetes.io/projected/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-kube-api-access-wh2st\") pod \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\" (UID: \"c5c39ff9-35bb-452d-94b3-b41fb9e16b70\") " Oct 31 00:40:31.207388 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.207363 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-openshift-ca" pod "c5c39ff9-35bb-452d-94b3-b41fb9e16b70" (UID: "c5c39ff9-35bb-452d-94b3-b41fb9e16b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:31.209036 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.209002 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-xks-ca" pod "c5c39ff9-35bb-452d-94b3-b41fb9e16b70" (UID: "c5c39ff9-35bb-452d-94b3-b41fb9e16b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:31.209119 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.209060 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-kube-api-access-wh2st" pod "c5c39ff9-35bb-452d-94b3-b41fb9e16b70" (UID: "c5c39ff9-35bb-452d-94b3-b41fb9e16b70"). InnerVolumeSpecName "kube-api-access-wh2st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:31.307972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.307937 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-xks-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:31.307972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.307971 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-maas-api-openshift-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:31.308145 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.307985 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2st\" (UniqueName: \"kubernetes.io/projected/c5c39ff9-35bb-452d-94b3-b41fb9e16b70-kube-api-access-wh2st\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:31.893632 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.893601 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" Oct 31 00:40:31.894125 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.893638 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz" event={"ID":"c5c39ff9-35bb-452d-94b3-b41fb9e16b70","Type":"ContainerDied","Data":"062732ff6c16710686cce27b408339e804b3d2869eeccf348028be12f7f4884c"} Oct 31 00:40:31.894125 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.893701 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="062732ff6c16710686cce27b408339e804b3d2869eeccf348028be12f7f4884c" Oct 31 00:40:31.903105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:31.902990 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:32.345714 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:32.345675 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9"] Oct 31 00:40:32.900476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:32.900443 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:32.901346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:32.901326 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:40:36.770350 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:36.770314 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q"] Oct 31 00:40:36.770857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:36.770577 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" podUID="176542b6-7815-4e3d-94c7-8fa987949922" containerName="maas-api" containerID="cri-o://5e53001d22efd895bb0a3012383b7abd2e31eff5d12232d1b681b64339a08f1c" gracePeriod=30 Oct 31 00:40:36.925366 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:36.925225 5890 generic.go:356] "Generic (PLEG): container finished" podID="176542b6-7815-4e3d-94c7-8fa987949922" containerID="5e53001d22efd895bb0a3012383b7abd2e31eff5d12232d1b681b64339a08f1c" exitCode=0 Oct 31 00:40:36.925366 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:36.925299 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" event={"ID":"176542b6-7815-4e3d-94c7-8fa987949922","Type":"ContainerDied","Data":"5e53001d22efd895bb0a3012383b7abd2e31eff5d12232d1b681b64339a08f1c"} Oct 31 00:40:37.020214 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:37.020175 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:40:37.042888 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.042866 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:40:37.075834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.075808 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-metrics-certs\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-metrics-certs\") pod \"176542b6-7815-4e3d-94c7-8fa987949922\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " Oct 31 00:40:37.075940 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.075893 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176542b6-7815-4e3d-94c7-8fa987949922-kube-api-access-5dlz6\" (UniqueName: \"kubernetes.io/projected/176542b6-7815-4e3d-94c7-8fa987949922-kube-api-access-5dlz6\") pod \"176542b6-7815-4e3d-94c7-8fa987949922\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " Oct 31 00:40:37.075940 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.075921 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176542b6-7815-4e3d-94c7-8fa987949922-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176542b6-7815-4e3d-94c7-8fa987949922-openshift-service-ca\") pod \"176542b6-7815-4e3d-94c7-8fa987949922\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " Oct 31 00:40:37.076039 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.075948 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls\") pod \"176542b6-7815-4e3d-94c7-8fa987949922\" (UID: \"176542b6-7815-4e3d-94c7-8fa987949922\") " Oct 31 00:40:37.076282 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.076254 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176542b6-7815-4e3d-94c7-8fa987949922-openshift-service-ca" pod "176542b6-7815-4e3d-94c7-8fa987949922" (UID: "176542b6-7815-4e3d-94c7-8fa987949922"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:37.077970 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.077941 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-metrics-certs" pod "176542b6-7815-4e3d-94c7-8fa987949922" (UID: "176542b6-7815-4e3d-94c7-8fa987949922"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:37.078117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.078020 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls" pod "176542b6-7815-4e3d-94c7-8fa987949922" (UID: "176542b6-7815-4e3d-94c7-8fa987949922"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:37.078117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.078042 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176542b6-7815-4e3d-94c7-8fa987949922-kube-api-access-5dlz6" pod "176542b6-7815-4e3d-94c7-8fa987949922" (UID: "176542b6-7815-4e3d-94c7-8fa987949922"). InnerVolumeSpecName "kube-api-access-5dlz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:37.177061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.177025 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:37.177061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.177055 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dlz6\" (UniqueName: \"kubernetes.io/projected/176542b6-7815-4e3d-94c7-8fa987949922-kube-api-access-5dlz6\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:37.177061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.177067 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176542b6-7815-4e3d-94c7-8fa987949922-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:37.177356 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.177076 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176542b6-7815-4e3d-94c7-8fa987949922-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:37.346140 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.346010 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" podUID="f04f3d86-2cda-4423-99ee-b26d45d25c70" containerName="payload-pre-processing" containerID="cri-o://6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed" gracePeriod=25 Oct 31 00:40:37.590091 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.590067 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:40:37.686064 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.686022 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04f3d86-2cda-4423-99ee-b26d45d25c70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04f3d86-2cda-4423-99ee-b26d45d25c70-plugins-config-volume\") pod \"f04f3d86-2cda-4423-99ee-b26d45d25c70\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " Oct 31 00:40:37.686274 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.686098 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04f3d86-2cda-4423-99ee-b26d45d25c70-kube-api-access-pxhdb\" (UniqueName: \"kubernetes.io/projected/f04f3d86-2cda-4423-99ee-b26d45d25c70-kube-api-access-pxhdb\") pod \"f04f3d86-2cda-4423-99ee-b26d45d25c70\" (UID: \"f04f3d86-2cda-4423-99ee-b26d45d25c70\") " Oct 31 00:40:37.687263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.687230 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04f3d86-2cda-4423-99ee-b26d45d25c70-plugins-config-volume" pod "f04f3d86-2cda-4423-99ee-b26d45d25c70" (UID: "f04f3d86-2cda-4423-99ee-b26d45d25c70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:37.689127 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.689086 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04f3d86-2cda-4423-99ee-b26d45d25c70-kube-api-access-pxhdb" pod "f04f3d86-2cda-4423-99ee-b26d45d25c70" (UID: "f04f3d86-2cda-4423-99ee-b26d45d25c70"). InnerVolumeSpecName "kube-api-access-pxhdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:37.787466 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.787439 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04f3d86-2cda-4423-99ee-b26d45d25c70-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:37.787466 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.787466 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxhdb\" (UniqueName: \"kubernetes.io/projected/f04f3d86-2cda-4423-99ee-b26d45d25c70-kube-api-access-pxhdb\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:37.940107 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.940020 5890 generic.go:356] "Generic (PLEG): container finished" podID="f04f3d86-2cda-4423-99ee-b26d45d25c70" containerID="6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed" exitCode=0 Oct 31 00:40:37.940276 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.940205 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" event={"ID":"f04f3d86-2cda-4423-99ee-b26d45d25c70","Type":"ContainerDied","Data":"6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed"} Oct 31 00:40:37.940276 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.940251 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" event={"ID":"f04f3d86-2cda-4423-99ee-b26d45d25c70","Type":"ContainerDied","Data":"e2bc9b33720bc7a7e6a83fff98686595c12e378dcdbc3ef21d879e1a9255b7b0"} Oct 31 00:40:37.940391 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.940281 5890 scope.go:122] "RemoveContainer" containerID="6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed" Oct 31 00:40:37.940445 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.940427 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9" Oct 31 00:40:37.943531 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.943503 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" event={"ID":"176542b6-7815-4e3d-94c7-8fa987949922","Type":"ContainerDied","Data":"09978401e13b156f2476f772feb90efd85e74f8e5051d3731ff4bd3301e32fe6"} Oct 31 00:40:37.943720 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.943581 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q" Oct 31 00:40:37.954441 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.954376 5890 scope.go:122] "RemoveContainer" containerID="6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed" Oct 31 00:40:37.954839 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:37.954769 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed\": container with ID starting with 6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed not found: ID does not exist" containerID="6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed" Oct 31 00:40:37.954934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.954852 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed"} err="failed to get container status \"6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed\": rpc error: code = NotFound desc = could not find container \"6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed\": container with ID starting with 6ba5e11e33e24e33067cf3f35c82c02b5054bead31e9ab6dc4e200672883a4ed not found: ID does not exist" Oct 31 00:40:37.954934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:37.954873 5890 scope.go:122] "RemoveContainer" containerID="5e53001d22efd895bb0a3012383b7abd2e31eff5d12232d1b681b64339a08f1c" Oct 31 00:40:38.103668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.103016 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9"] Oct 31 00:40:38.106582 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.106553 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-24d78330-7d4499b8ff-6z4t9"] Oct 31 00:40:38.210466 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.210392 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q"] Oct 31 00:40:38.215757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.215712 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24d78330-6548fdbd84-pbr4q"] Oct 31 00:40:38.730135 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730046 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f"] Oct 31 00:40:38.730637 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730618 5890 cpu_manager.go:414] "containerMap: removing container" podUID="f04f3d86-2cda-4423-99ee-b26d45d25c70" containerName="payload-pre-processing" Oct 31 00:40:38.730759 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730638 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04f3d86-2cda-4423-99ee-b26d45d25c70" containerName="payload-pre-processing" Oct 31 00:40:38.730759 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730651 5890 cpu_manager.go:414] "containerMap: removing container" podUID="176542b6-7815-4e3d-94c7-8fa987949922" containerName="maas-api" Oct 31 00:40:38.730759 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730658 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176542b6-7815-4e3d-94c7-8fa987949922" containerName="maas-api" Oct 31 00:40:38.730759 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730713 5890 cpu_manager.go:414] "containerMap: removing container" podUID="c5c39ff9-35bb-452d-94b3-b41fb9e16b70" containerName="revoke-keys" Oct 31 00:40:38.730759 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730723 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c39ff9-35bb-452d-94b3-b41fb9e16b70" containerName="revoke-keys" Oct 31 00:40:38.731037 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730838 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c39ff9-35bb-452d-94b3-b41fb9e16b70" containerName="revoke-keys" Oct 31 00:40:38.731037 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730853 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04f3d86-2cda-4423-99ee-b26d45d25c70" containerName="payload-pre-processing" Oct 31 00:40:38.731037 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.730864 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="176542b6-7815-4e3d-94c7-8fa987949922" containerName="maas-api" Oct 31 00:40:38.733859 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.733838 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:38.802813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.790143 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-29ece5\"" Oct 31 00:40:38.802813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.790287 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-29ece5-dockercfg-bw89t\"" Oct 31 00:40:38.802813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.801632 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hh4d\" (UniqueName: \"kubernetes.io/projected/71d79d95-9651-4aed-a942-e4d087b335d5-kube-api-access-7hh4d\") pod \"payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:38.802813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.801698 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d79d95-9651-4aed-a942-e4d087b335d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:38.815049 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.815018 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f"] Oct 31 00:40:38.837276 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.837244 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf"] Oct 31 00:40:38.839812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.839770 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:38.896601 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.896461 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf"] Oct 31 00:40:38.902388 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.902352 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hh4d\" (UniqueName: \"kubernetes.io/projected/71d79d95-9651-4aed-a942-e4d087b335d5-kube-api-access-7hh4d\") pod \"payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:38.902551 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.902434 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjqg\" (UniqueName: \"kubernetes.io/projected/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-kube-api-access-vhjqg\") pod \"payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:38.902551 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.902468 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d79d95-9651-4aed-a942-e4d087b335d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:38.902674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.902565 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:38.903369 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.903346 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d79d95-9651-4aed-a942-e4d087b335d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:38.999398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:38.999279 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hh4d\" (UniqueName: \"kubernetes.io/projected/71d79d95-9651-4aed-a942-e4d087b335d5-kube-api-access-7hh4d\") pod \"payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:39.005537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.004324 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:39.005537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.004484 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjqg\" (UniqueName: \"kubernetes.io/projected/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-kube-api-access-vhjqg\") pod \"payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:39.005537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.005396 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:39.046772 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.046724 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:39.065843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.065816 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjqg\" (UniqueName: \"kubernetes.io/projected/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-kube-api-access-vhjqg\") pod \"payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:39.155457 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.155426 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:39.417967 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.417936 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f"] Oct 31 00:40:39.418129 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:39.418082 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d79d95_9651_4aed_a942_e4d087b335d5.slice/crio-f11962105be5960017d5867e6fe07dbcbf33a4c00938d267d088662d5113d860 WatchSource:0}: Error finding container f11962105be5960017d5867e6fe07dbcbf33a4c00938d267d088662d5113d860: Status 404 returned error can't find the container with id f11962105be5960017d5867e6fe07dbcbf33a4c00938d267d088662d5113d860 Oct 31 00:40:39.539868 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.539776 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf"] Oct 31 00:40:39.932294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.932260 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176542b6-7815-4e3d-94c7-8fa987949922" path="/var/lib/kubelet/pods/176542b6-7815-4e3d-94c7-8fa987949922/volumes" Oct 31 00:40:39.932666 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.932627 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04f3d86-2cda-4423-99ee-b26d45d25c70" path="/var/lib/kubelet/pods/f04f3d86-2cda-4423-99ee-b26d45d25c70/volumes" Oct 31 00:40:39.959348 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.959312 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" event={"ID":"71d79d95-9651-4aed-a942-e4d087b335d5","Type":"ContainerStarted","Data":"c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b"} Oct 31 00:40:39.959491 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.959366 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:39.959491 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.959385 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" event={"ID":"71d79d95-9651-4aed-a942-e4d087b335d5","Type":"ContainerStarted","Data":"f11962105be5960017d5867e6fe07dbcbf33a4c00938d267d088662d5113d860"} Oct 31 00:40:39.960880 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.960854 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" event={"ID":"3c4e2839-34eb-4d91-aaa2-969f5dea68d8","Type":"ContainerStarted","Data":"5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1"} Oct 31 00:40:39.961015 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.960898 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" event={"ID":"3c4e2839-34eb-4d91-aaa2-969f5dea68d8","Type":"ContainerStarted","Data":"5eac4c743a80733f7e542ed6c08727b177c9afe064f20f723dfc2e7bd1800288"} Oct 31 00:40:39.961015 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:39.960926 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:40.086778 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:40.086700 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" podStartSLOduration=2.086685656 podStartE2EDuration="2.086685656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:40.08495029 +0000 UTC m=+732.287200727" watchObservedRunningTime="2026-10-31 00:40:40.086685656 +0000 UTC m=+732.288936083" Oct 31 00:40:40.197549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:40.196530 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" podStartSLOduration=2.196511216 podStartE2EDuration="2.196511216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:40.195754743 +0000 UTC m=+732.398005178" watchObservedRunningTime="2026-10-31 00:40:40.196511216 +0000 UTC m=+732.398761651" Oct 31 00:40:40.794023 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:40.793980 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z"] Oct 31 00:40:40.794252 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:40.794226 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" containerID="cri-o://a0cc58e3f00f19a2565c4432bb99b65df63f4eb4c2fcfae121de0708560a1173" gracePeriod=30 Oct 31 00:40:42.308802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:42.308752 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6"] Oct 31 00:40:42.309295 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:42.309045 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" containerID="cri-o://fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794" gracePeriod=30 Oct 31 00:40:45.998277 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:45.998244 5890 generic.go:356] "Generic (PLEG): container finished" podID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerID="a0cc58e3f00f19a2565c4432bb99b65df63f4eb4c2fcfae121de0708560a1173" exitCode=0 Oct 31 00:40:45.998678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:45.998321 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" event={"ID":"e0dc4d09-1427-4a19-9903-597de4b83e45","Type":"ContainerDied","Data":"a0cc58e3f00f19a2565c4432bb99b65df63f4eb4c2fcfae121de0708560a1173"} Oct 31 00:40:45.998678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:45.998400 5890 scope.go:122] "RemoveContainer" containerID="8b9b9c872e44cd46cfe1eda417751d832093d5ddea65532da7e533e79fc9e1b6" Oct 31 00:40:46.066096 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.066034 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:40:46.095067 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095035 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-data\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095090 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-token\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-token\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095110 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-podinfo\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095185 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-credential-socket\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095212 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-envoy\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095233 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0dc4d09-1427-4a19-9903-597de4b83e45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0dc4d09-1427-4a19-9903-597de4b83e45-istiod-ca-cert\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095301 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-kube-api-access-v59cw\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-kube-api-access-v59cw\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095473 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095338 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-certs\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.095473 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095353 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-data" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:46.095473 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095379 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-socket\") pod \"e0dc4d09-1427-4a19-9903-597de4b83e45\" (UID: \"e0dc4d09-1427-4a19-9903-597de4b83e45\") " Oct 31 00:40:46.096721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095498 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-credential-socket" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:46.096721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095863 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.096721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095914 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.096721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095930 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dc4d09-1427-4a19-9903-597de4b83e45-istiod-ca-cert" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:46.096721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.095960 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-certs" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:46.096721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.096013 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-socket" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:46.097601 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.097577 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-podinfo" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:40:46.097827 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.097756 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-envoy" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:46.097910 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.097844 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-token" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:46.098227 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.098208 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-kube-api-access-v59cw" pod "e0dc4d09-1427-4a19-9903-597de4b83e45" (UID: "e0dc4d09-1427-4a19-9903-597de4b83e45"). InnerVolumeSpecName "kube-api-access-v59cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:46.197065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.197034 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.197065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.197062 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0dc4d09-1427-4a19-9903-597de4b83e45-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.197065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.197072 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59cw\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-kube-api-access-v59cw\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.197294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.197083 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.197294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.197092 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0dc4d09-1427-4a19-9903-597de4b83e45-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.197294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.197101 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.197294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.197110 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0dc4d09-1427-4a19-9903-597de4b83e45-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.655475 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.655423 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4"] Oct 31 00:40:46.657890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.657866 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30d0c8aa-vb8c4"] Oct 31 00:40:46.732476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.732432 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv"] Oct 31 00:40:46.732711 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:46.732684 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" containerID="cri-o://bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046" gracePeriod=30 Oct 31 00:40:47.004837 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.004803 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" event={"ID":"e0dc4d09-1427-4a19-9903-597de4b83e45","Type":"ContainerDied","Data":"a3dcb61fd9e18746ddad391a90d79ae687a9d2bb88c66d46c42da0507dd64574"} Oct 31 00:40:47.005288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.004864 5890 scope.go:122] "RemoveContainer" containerID="a0cc58e3f00f19a2565c4432bb99b65df63f4eb4c2fcfae121de0708560a1173" Oct 31 00:40:47.005288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.004821 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z" Oct 31 00:40:47.164347 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.164316 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z"] Oct 31 00:40:47.169378 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.169350 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-24d78330-openshift-default-f7bc5cf4c-tq64z"] Oct 31 00:40:47.577880 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.577825 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:40:47.609863 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.609836 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-certs\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.609984 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.609867 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-socket\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.609984 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.609904 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd118214-d05a-4dc1-9177-7d998352f52f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd118214-d05a-4dc1-9177-7d998352f52f-istiod-ca-cert\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.609984 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.609932 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-envoy\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.610095 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.609991 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-data\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.610095 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610018 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-kube-api-access-znvpn\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-kube-api-access-znvpn\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.610095 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610038 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-credential-socket\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.610095 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610058 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-istio-token\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-istio-token\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.610302 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610106 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd118214-d05a-4dc1-9177-7d998352f52f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd118214-d05a-4dc1-9177-7d998352f52f-istio-podinfo\") pod \"fd118214-d05a-4dc1-9177-7d998352f52f\" (UID: \"fd118214-d05a-4dc1-9177-7d998352f52f\") " Oct 31 00:40:47.610302 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610132 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-certs" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:47.610302 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610143 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-socket" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:47.610460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610381 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-credential-socket" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:47.610460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610398 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-data" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:47.610460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610429 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd118214-d05a-4dc1-9177-7d998352f52f-istiod-ca-cert" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:47.610625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610535 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.610625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610554 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.610625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610568 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd118214-d05a-4dc1-9177-7d998352f52f-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.610625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610582 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.610625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.610595 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.612207 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.612185 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-kube-api-access-znvpn" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "kube-api-access-znvpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:47.612385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.612364 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-istio-token" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:47.612608 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.612585 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-envoy" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:47.612691 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.612605 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd118214-d05a-4dc1-9177-7d998352f52f-istio-podinfo" pod "fd118214-d05a-4dc1-9177-7d998352f52f" (UID: "fd118214-d05a-4dc1-9177-7d998352f52f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:40:47.711383 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.711348 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd118214-d05a-4dc1-9177-7d998352f52f-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.711383 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.711382 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znvpn\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-kube-api-access-znvpn\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.711568 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.711397 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd118214-d05a-4dc1-9177-7d998352f52f-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.711568 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.711410 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd118214-d05a-4dc1-9177-7d998352f52f-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:47.931729 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.931697 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d86a92-b73d-46ce-9537-1cf7da92821f" path="/var/lib/kubelet/pods/90d86a92-b73d-46ce-9537-1cf7da92821f/volumes" Oct 31 00:40:47.932172 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:47.932157 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" path="/var/lib/kubelet/pods/e0dc4d09-1427-4a19-9903-597de4b83e45/volumes" Oct 31 00:40:48.014816 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.014757 5890 generic.go:356] "Generic (PLEG): container finished" podID="fd118214-d05a-4dc1-9177-7d998352f52f" containerID="fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794" exitCode=0 Oct 31 00:40:48.015307 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.014995 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" Oct 31 00:40:48.016408 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.015500 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" event={"ID":"fd118214-d05a-4dc1-9177-7d998352f52f","Type":"ContainerDied","Data":"fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794"} Oct 31 00:40:48.016408 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.015559 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6" event={"ID":"fd118214-d05a-4dc1-9177-7d998352f52f","Type":"ContainerDied","Data":"dd4d8ffdcb9fe06d7653d44af16ce82bc2f3b7262bf434334ef8550d683ebd51"} Oct 31 00:40:48.016408 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.015599 5890 scope.go:122] "RemoveContainer" containerID="fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794" Oct 31 00:40:48.032919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.032867 5890 scope.go:122] "RemoveContainer" containerID="d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954" Oct 31 00:40:48.042385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.042361 5890 scope.go:122] "RemoveContainer" containerID="fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794" Oct 31 00:40:48.042669 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:48.042647 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794\": container with ID starting with fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794 not found: ID does not exist" containerID="fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794" Oct 31 00:40:48.042813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.042683 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794"} err="failed to get container status \"fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794\": rpc error: code = NotFound desc = could not find container \"fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794\": container with ID starting with fa8b39036c311a297a48a22eee52c64a49384d518d54921c0d34d7ad6caeb794 not found: ID does not exist" Oct 31 00:40:48.042813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.042708 5890 scope.go:122] "RemoveContainer" containerID="d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954" Oct 31 00:40:48.043079 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:48.043062 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954\": container with ID starting with d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954 not found: ID does not exist" containerID="d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954" Oct 31 00:40:48.043124 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.043086 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954"} err="failed to get container status \"d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954\": rpc error: code = NotFound desc = could not find container \"d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954\": container with ID starting with d927b924c887d3aca28ffb038d8ab4ed78f21aba4c5257aca7f276466e2fb954 not found: ID does not exist" Oct 31 00:40:48.113598 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.113564 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp"] Oct 31 00:40:48.114043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114028 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" Oct 31 00:40:48.114097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114046 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" Oct 31 00:40:48.114097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114059 5890 cpu_manager.go:414] "containerMap: removing container" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" Oct 31 00:40:48.114097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114067 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" Oct 31 00:40:48.114097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114077 5890 cpu_manager.go:414] "containerMap: removing container" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" Oct 31 00:40:48.114097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114083 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" Oct 31 00:40:48.114249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114193 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" Oct 31 00:40:48.114249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114205 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" Oct 31 00:40:48.114249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114215 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" Oct 31 00:40:48.114249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.114223 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" containerName="istio-proxy" Oct 31 00:40:48.118978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.118951 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.169368 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.169338 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd98f0-openshift-default-dockercfg-6r29c\"" Oct 31 00:40:48.180535 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.180506 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp"] Oct 31 00:40:48.184092 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.184013 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6"] Oct 31 00:40:48.186467 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.186442 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2b9202f4-openshift-default-6c7dcff65-rgns6"] Oct 31 00:40:48.215261 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215230 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215449 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215270 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxt5\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-kube-api-access-8rxt5\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215449 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215292 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215449 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215412 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215450 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467d4202-b55f-4b78-8f8c-07184fbb79b1-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215517 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215535 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215577 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.215607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.215602 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.316401 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316367 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.316609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316408 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.316609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316437 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467d4202-b55f-4b78-8f8c-07184fbb79b1-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.316609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316503 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.316609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316525 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.316609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316582 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.317142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316619 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.317142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316651 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.317142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316684 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxt5\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-kube-api-access-8rxt5\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.317142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.316939 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.317729 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.317504 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.317729 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.317544 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.317729 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.317677 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.318019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.317988 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467d4202-b55f-4b78-8f8c-07184fbb79b1-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.319422 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.319400 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.319529 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.319495 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.385174 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.385101 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.394352 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.394318 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxt5\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-kube-api-access-8rxt5\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.431358 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.431323 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:48.853760 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.853408 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp"] Oct 31 00:40:48.855183 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:48.855153 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467d4202_b55f_4b78_8f8c_07184fbb79b1.slice/crio-d5e2e1fcb9452512eb3f25dba1bc5ca131a2fd0fc903568e65bc22b85312b8ec WatchSource:0}: Error finding container d5e2e1fcb9452512eb3f25dba1bc5ca131a2fd0fc903568e65bc22b85312b8ec: Status 404 returned error can't find the container with id d5e2e1fcb9452512eb3f25dba1bc5ca131a2fd0fc903568e65bc22b85312b8ec Oct 31 00:40:48.858332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.858295 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:48.858420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.858377 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:48.858472 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:48.858416 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:49.029268 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:49.029214 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" event={"ID":"467d4202-b55f-4b78-8f8c-07184fbb79b1","Type":"ContainerStarted","Data":"b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616"} Oct 31 00:40:49.029713 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:49.029276 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" event={"ID":"467d4202-b55f-4b78-8f8c-07184fbb79b1","Type":"ContainerStarted","Data":"d5e2e1fcb9452512eb3f25dba1bc5ca131a2fd0fc903568e65bc22b85312b8ec"} Oct 31 00:40:49.147050 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:49.146947 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podStartSLOduration=1.146931777 podStartE2EDuration="1.146931777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:49.145553655 +0000 UTC m=+741.347804089" watchObservedRunningTime="2026-10-31 00:40:49.146931777 +0000 UTC m=+741.349182189" Oct 31 00:40:49.431767 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:49.431734 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:49.437220 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:49.437191 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:49.934951 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:49.934918 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd118214-d05a-4dc1-9177-7d998352f52f" path="/var/lib/kubelet/pods/fd118214-d05a-4dc1-9177-7d998352f52f/volumes" Oct 31 00:40:50.035080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:50.035045 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:50.036053 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:50.036028 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:40:50.969348 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:50.969322 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 00:40:50.969540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:50.969384 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 00:40:51.396354 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.396276 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc"] Oct 31 00:40:51.396713 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.396703 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" Oct 31 00:40:51.396754 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.396715 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0dc4d09-1427-4a19-9903-597de4b83e45" containerName="istio-proxy" Oct 31 00:40:51.401038 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.401020 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.446993 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.446958 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4llxc\" (UniqueName: \"kubernetes.io/projected/8a6b1212-8008-4f0e-a86f-638da400aeea-kube-api-access-4llxc\") pod \"payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.447159 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.447024 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b1212-8008-4f0e-a86f-638da400aeea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.451888 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.451860 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ef2572\"" Oct 31 00:40:51.451888 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.451860 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ef2572-dockercfg-thpd2\"" Oct 31 00:40:51.454662 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.454635 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc"] Oct 31 00:40:51.506551 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.506507 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r"] Oct 31 00:40:51.516445 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.516413 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.548187 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.548155 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b0b7a9-4e13-42a7-809f-360661bb960e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.548348 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.548206 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b1212-8008-4f0e-a86f-638da400aeea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.548421 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.548398 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4llxc\" (UniqueName: \"kubernetes.io/projected/8a6b1212-8008-4f0e-a86f-638da400aeea-kube-api-access-4llxc\") pod \"payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.548458 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.548446 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2td\" (UniqueName: \"kubernetes.io/projected/b6b0b7a9-4e13-42a7-809f-360661bb960e-kube-api-access-zl2td\") pod \"payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.548969 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.548954 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b1212-8008-4f0e-a86f-638da400aeea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.561985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.561956 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r"] Oct 31 00:40:51.606705 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.606675 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4llxc\" (UniqueName: \"kubernetes.io/projected/8a6b1212-8008-4f0e-a86f-638da400aeea-kube-api-access-4llxc\") pod \"payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.649427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.649342 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2td\" (UniqueName: \"kubernetes.io/projected/b6b0b7a9-4e13-42a7-809f-360661bb960e-kube-api-access-zl2td\") pod \"payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.649427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.649407 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b0b7a9-4e13-42a7-809f-360661bb960e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.650196 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.650175 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b0b7a9-4e13-42a7-809f-360661bb960e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.708282 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.708250 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2td\" (UniqueName: \"kubernetes.io/projected/b6b0b7a9-4e13-42a7-809f-360661bb960e-kube-api-access-zl2td\") pod \"payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.714950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.714926 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:51.830833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.830799 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:51.995331 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:51.995305 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:40:52.050484 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.050452 5890 generic.go:356] "Generic (PLEG): container finished" podID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerID="bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046" exitCode=0 Oct 31 00:40:52.050661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.050517 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" Oct 31 00:40:52.050661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.050542 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" event={"ID":"0562ef5c-d803-4a72-9597-76ebcdaff508","Type":"ContainerDied","Data":"bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046"} Oct 31 00:40:52.050661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.050610 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv" event={"ID":"0562ef5c-d803-4a72-9597-76ebcdaff508","Type":"ContainerDied","Data":"595290fb0eede028fa7fed514873c826d0026e1f5d0da978086d41fdfe512909"} Oct 31 00:40:52.050661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.050639 5890 scope.go:122] "RemoveContainer" containerID="bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046" Oct 31 00:40:52.052961 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.052929 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0562ef5c-d803-4a72-9597-76ebcdaff508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562ef5c-d803-4a72-9597-76ebcdaff508-istiod-ca-cert\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053057 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.052972 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-envoy\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053057 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053029 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-podinfo\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053057 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053053 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-data\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053222 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053082 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-credential-socket\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053222 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053129 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-certs\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053222 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053180 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-socket\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053252 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-token\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-token\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053299 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-kube-api-access-ff42s\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-kube-api-access-ff42s\") pod \"0562ef5c-d803-4a72-9597-76ebcdaff508\" (UID: \"0562ef5c-d803-4a72-9597-76ebcdaff508\") " Oct 31 00:40:52.053379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053308 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0562ef5c-d803-4a72-9597-76ebcdaff508-istiod-ca-cert" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:52.053610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053579 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-certs" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:52.053610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053596 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-socket" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:52.053751 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053727 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-data" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:52.053751 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053742 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-credential-socket" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:52.053881 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053843 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.053881 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053867 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.053977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.053882 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562ef5c-d803-4a72-9597-76ebcdaff508-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.055701 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.055659 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-envoy" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:40:52.055822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.055754 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-podinfo" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:40:52.055980 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.055959 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-kube-api-access-ff42s" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "kube-api-access-ff42s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:52.057168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.057144 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-token" pod "0562ef5c-d803-4a72-9597-76ebcdaff508" (UID: "0562ef5c-d803-4a72-9597-76ebcdaff508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:52.061936 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.061918 5890 scope.go:122] "RemoveContainer" containerID="926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07" Oct 31 00:40:52.070580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.070554 5890 scope.go:122] "RemoveContainer" containerID="bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046" Oct 31 00:40:52.070909 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:52.070887 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046\": container with ID starting with bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046 not found: ID does not exist" containerID="bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046" Oct 31 00:40:52.071010 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.070914 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046"} err="failed to get container status \"bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046\": rpc error: code = NotFound desc = could not find container \"bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046\": container with ID starting with bcc4be09771b2548f8bdec2cec34d9f6c25ac99219651b785b2f2335ecd25046 not found: ID does not exist" Oct 31 00:40:52.071010 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.070930 5890 scope.go:122] "RemoveContainer" containerID="926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07" Oct 31 00:40:52.071235 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:52.071189 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07\": container with ID starting with 926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07 not found: ID does not exist" containerID="926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07" Oct 31 00:40:52.071305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.071225 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07"} err="failed to get container status \"926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07\": rpc error: code = NotFound desc = could not find container \"926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07\": container with ID starting with 926f7a300354451afe115bccbfd26c32ec316825399e06c0ff0dcafce52e9a07 not found: ID does not exist" Oct 31 00:40:52.093594 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.093572 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc"] Oct 31 00:40:52.094878 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:52.094857 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6b1212_8008_4f0e_a86f_638da400aeea.slice/crio-8942e3edf8d1780528fba5c57087cf05391e29fa4d051f1465a52e68e3461df5 WatchSource:0}: Error finding container 8942e3edf8d1780528fba5c57087cf05391e29fa4d051f1465a52e68e3461df5: Status 404 returned error can't find the container with id 8942e3edf8d1780528fba5c57087cf05391e29fa4d051f1465a52e68e3461df5 Oct 31 00:40:52.154938 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.154908 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.155099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.155042 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.155099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.155066 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.155099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.155075 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.155099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.155085 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff42s\" (UniqueName: \"kubernetes.io/projected/0562ef5c-d803-4a72-9597-76ebcdaff508-kube-api-access-ff42s\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.155099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.155097 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562ef5c-d803-4a72-9597-76ebcdaff508-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.200625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.200597 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r"] Oct 31 00:40:52.203556 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:52.203529 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b0b7a9_4e13_42a7_809f_360661bb960e.slice/crio-48287d6ed431252dcea87f05d292701f643fb78018767a3607a83eb9ca81bf98 WatchSource:0}: Error finding container 48287d6ed431252dcea87f05d292701f643fb78018767a3607a83eb9ca81bf98: Status 404 returned error can't find the container with id 48287d6ed431252dcea87f05d292701f643fb78018767a3607a83eb9ca81bf98 Oct 31 00:40:52.471257 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.471184 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv"] Oct 31 00:40:52.476757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:52.476730 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-30d0c8aa-gw-openshift-default-6b576f5fb5-rqjzv"] Oct 31 00:40:53.058099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.058056 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" event={"ID":"8a6b1212-8008-4f0e-a86f-638da400aeea","Type":"ContainerStarted","Data":"4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00"} Oct 31 00:40:53.058294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.058252 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:40:53.058345 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.058296 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" event={"ID":"8a6b1212-8008-4f0e-a86f-638da400aeea","Type":"ContainerStarted","Data":"8942e3edf8d1780528fba5c57087cf05391e29fa4d051f1465a52e68e3461df5"} Oct 31 00:40:53.060591 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.060560 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" event={"ID":"b6b0b7a9-4e13-42a7-809f-360661bb960e","Type":"ContainerStarted","Data":"464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2"} Oct 31 00:40:53.060735 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.060603 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" event={"ID":"b6b0b7a9-4e13-42a7-809f-360661bb960e","Type":"ContainerStarted","Data":"48287d6ed431252dcea87f05d292701f643fb78018767a3607a83eb9ca81bf98"} Oct 31 00:40:53.060735 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.060664 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:40:53.169703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.169652 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" podStartSLOduration=2.16963991 podStartE2EDuration="2.16963991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:53.16921704 +0000 UTC m=+745.371467474" watchObservedRunningTime="2026-10-31 00:40:53.16963991 +0000 UTC m=+745.371890342" Oct 31 00:40:53.282137 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.282064 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" podStartSLOduration=2.282045571 podStartE2EDuration="2.282045571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:53.280593836 +0000 UTC m=+745.482844272" watchObservedRunningTime="2026-10-31 00:40:53.282045571 +0000 UTC m=+745.484296005" Oct 31 00:40:53.932401 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:53.932369 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" path="/var/lib/kubelet/pods/0562ef5c-d803-4a72-9597-76ebcdaff508/volumes" Oct 31 00:40:57.856838 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.852472 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57"] Oct 31 00:40:57.868807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.867101 5890 cpu_manager.go:414] "containerMap: removing container" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" Oct 31 00:40:57.868807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.867130 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" Oct 31 00:40:57.868807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.867150 5890 cpu_manager.go:414] "containerMap: removing container" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" Oct 31 00:40:57.868807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.867159 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" Oct 31 00:40:57.868807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.867287 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" Oct 31 00:40:57.868807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.867301 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="0562ef5c-d803-4a72-9597-76ebcdaff508" containerName="istio-proxy" Oct 31 00:40:57.871812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.871052 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:57.911370 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.911337 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57"] Oct 31 00:40:57.915213 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.915183 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhhs\" (UniqueName: \"kubernetes.io/projected/2498dc56-4e7a-45f0-bc55-ce80a12e774b-kube-api-access-jbhhs\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:57.915379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.915267 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-metrics-certs\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:57.915457 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.915410 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2498dc56-4e7a-45f0-bc55-ce80a12e774b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:57.915536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.915514 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:57.968547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.968074 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ef2572-serving-cert\"" Oct 31 00:40:57.968547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.968220 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:40:57.968547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.968309 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f72zx\"" Oct 31 00:40:57.968547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.968427 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:40:57.968547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:57.968519 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:40:58.016805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.016753 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.016950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.016851 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhhs\" (UniqueName: \"kubernetes.io/projected/2498dc56-4e7a-45f0-bc55-ce80a12e774b-kube-api-access-jbhhs\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.016950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.016929 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-metrics-certs\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.017055 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:58.016962 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-serving-cert: secret "maas-api-e2e-worker-w4-ef2572-serving-cert" not found Oct 31 00:40:58.017055 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:40:58.017027 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls podName:2498dc56-4e7a-45f0-bc55-ce80a12e774b nodeName:}" failed. No retries permitted until 2026-10-31 00:40:58.517007469 +0000 UTC m=+750.719257898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls") pod "maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" (UID: "2498dc56-4e7a-45f0-bc55-ce80a12e774b") : secret "maas-api-e2e-worker-w4-ef2572-serving-cert" not found Oct 31 00:40:58.017055 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.017047 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2498dc56-4e7a-45f0-bc55-ce80a12e774b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.017716 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.017689 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2498dc56-4e7a-45f0-bc55-ce80a12e774b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.019306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.019286 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-metrics-certs\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.076263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.076228 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhhs\" (UniqueName: \"kubernetes.io/projected/2498dc56-4e7a-45f0-bc55-ce80a12e774b-kube-api-access-jbhhs\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.378803 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.378735 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz"] Oct 31 00:40:58.381794 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.381758 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-95fa4445-pblpz"] Oct 31 00:40:58.489758 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.489717 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl"] Oct 31 00:40:58.492522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.492498 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:58.526673 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.526648 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.528998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.528979 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.557145 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.557113 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b63c24\"" Oct 31 00:40:58.571824 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.561006 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl"] Oct 31 00:40:58.572095 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.564478 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b63c24-dockercfg-whht5\"" Oct 31 00:40:58.627647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.627613 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d67e149-e4ad-489b-b42f-f53232be9311-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:58.627853 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.627649 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncpm9\" (UniqueName: \"kubernetes.io/projected/7d67e149-e4ad-489b-b42f-f53232be9311-kube-api-access-ncpm9\") pod \"payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:58.729165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.729127 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d67e149-e4ad-489b-b42f-f53232be9311-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:58.729165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.729166 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncpm9\" (UniqueName: \"kubernetes.io/projected/7d67e149-e4ad-489b-b42f-f53232be9311-kube-api-access-ncpm9\") pod \"payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:58.729883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.729862 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d67e149-e4ad-489b-b42f-f53232be9311-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:58.789439 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.789405 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:40:58.790397 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.790366 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncpm9\" (UniqueName: \"kubernetes.io/projected/7d67e149-e4ad-489b-b42f-f53232be9311-kube-api-access-ncpm9\") pod \"payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:58.805201 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:58.805177 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:40:59.272675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:59.272551 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl"] Oct 31 00:40:59.276907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:59.276866 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57"] Oct 31 00:40:59.316129 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:40:59.316092 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2498dc56_4e7a_45f0_bc55_ce80a12e774b.slice/crio-41524ee90e97ddd92522cadf32a1d2f6b5842d15042b3c736ba850b502a80700 WatchSource:0}: Error finding container 41524ee90e97ddd92522cadf32a1d2f6b5842d15042b3c736ba850b502a80700: Status 404 returned error can't find the container with id 41524ee90e97ddd92522cadf32a1d2f6b5842d15042b3c736ba850b502a80700 Oct 31 00:40:59.932365 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:40:59.932330 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c39ff9-35bb-452d-94b3-b41fb9e16b70" path="/var/lib/kubelet/pods/c5c39ff9-35bb-452d-94b3-b41fb9e16b70/volumes" Oct 31 00:41:00.113050 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.113014 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" event={"ID":"7d67e149-e4ad-489b-b42f-f53232be9311","Type":"ContainerStarted","Data":"5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df"} Oct 31 00:41:00.113321 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.113078 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" event={"ID":"7d67e149-e4ad-489b-b42f-f53232be9311","Type":"ContainerStarted","Data":"89e8ab67363c09bfbf7c37f453abac085d7f6f0e3a500de3d1e70842f75e1211"} Oct 31 00:41:00.113321 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.113110 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:41:00.114382 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.114358 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" event={"ID":"2498dc56-4e7a-45f0-bc55-ce80a12e774b","Type":"ContainerStarted","Data":"41524ee90e97ddd92522cadf32a1d2f6b5842d15042b3c736ba850b502a80700"} Oct 31 00:41:00.227409 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.227319 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" podStartSLOduration=2.22730567 podStartE2EDuration="2.22730567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:00.226255136 +0000 UTC m=+752.428505569" watchObservedRunningTime="2026-10-31 00:41:00.22730567 +0000 UTC m=+752.429556113" Oct 31 00:41:00.736025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.735994 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc"] Oct 31 00:41:00.738643 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.738626 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:00.795022 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.794088 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-dd98f0-dockercfg-6zqrd\"" Oct 31 00:41:00.797667 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.797636 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc"] Oct 31 00:41:00.841578 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.841542 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-dd98f0\"" Oct 31 00:41:00.879851 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.879818 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075ca02a-e1be-4229-997a-f1bdc5fbcb31-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:00.880005 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.879868 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnxnc\" (UniqueName: \"kubernetes.io/projected/075ca02a-e1be-4229-997a-f1bdc5fbcb31-kube-api-access-dnxnc\") pod \"payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:00.981420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.981383 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075ca02a-e1be-4229-997a-f1bdc5fbcb31-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:00.981580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.981454 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnxnc\" (UniqueName: \"kubernetes.io/projected/075ca02a-e1be-4229-997a-f1bdc5fbcb31-kube-api-access-dnxnc\") pod \"payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:00.982672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:00.982647 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075ca02a-e1be-4229-997a-f1bdc5fbcb31-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:01.040503 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:01.040434 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnxnc\" (UniqueName: \"kubernetes.io/projected/075ca02a-e1be-4229-997a-f1bdc5fbcb31-kube-api-access-dnxnc\") pod \"payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:01.072014 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:01.071970 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:01.123989 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:01.123958 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" event={"ID":"2498dc56-4e7a-45f0-bc55-ce80a12e774b","Type":"ContainerStarted","Data":"717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4"} Oct 31 00:41:01.124292 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:01.124270 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:41:01.270953 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:01.270892 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" podStartSLOduration=3.357268046 podStartE2EDuration="4.27087763s" totalImagesPullingTime="913.609584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:01.269060901 +0000 UTC m=+753.471311337" watchObservedRunningTime="2026-10-31 00:41:01.27087763 +0000 UTC m=+753.473128097" Oct 31 00:41:01.460413 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:01.460363 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc"] Oct 31 00:41:01.473455 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:41:01.473420 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075ca02a_e1be_4229_997a_f1bdc5fbcb31.slice/crio-9ce9165ca1a400b533c67190ea42898aff08a3d3aaf93d8f575dd8ed2aa24dcb WatchSource:0}: Error finding container 9ce9165ca1a400b533c67190ea42898aff08a3d3aaf93d8f575dd8ed2aa24dcb: Status 404 returned error can't find the container with id 9ce9165ca1a400b533c67190ea42898aff08a3d3aaf93d8f575dd8ed2aa24dcb Oct 31 00:41:02.130950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:02.130904 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" event={"ID":"075ca02a-e1be-4229-997a-f1bdc5fbcb31","Type":"ContainerStarted","Data":"edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437"} Oct 31 00:41:02.131398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:02.131105 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:02.131398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:02.131125 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" event={"ID":"075ca02a-e1be-4229-997a-f1bdc5fbcb31","Type":"ContainerStarted","Data":"9ce9165ca1a400b533c67190ea42898aff08a3d3aaf93d8f575dd8ed2aa24dcb"} Oct 31 00:41:02.287503 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:02.287382 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" podStartSLOduration=2.287360878 podStartE2EDuration="2.287360878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:02.285619155 +0000 UTC m=+754.487869588" watchObservedRunningTime="2026-10-31 00:41:02.287360878 +0000 UTC m=+754.489611312" Oct 31 00:41:03.942946 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:03.942900 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:41:03.943456 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:03.942967 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:04.071411 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:04.071377 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 00:41:04.071587 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:04.071463 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 00:41:06.162731 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:06.162698 5890 generic.go:356] "Generic (PLEG): container finished" podID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerID="0ebd8a6ccebe6eb6443fe36868eceec141883f3c653882c9aab600050d0d4015" exitCode=0 Oct 31 00:41:06.163194 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:06.162776 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" event={"ID":"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7","Type":"ContainerDied","Data":"0ebd8a6ccebe6eb6443fe36868eceec141883f3c653882c9aab600050d0d4015"} Oct 31 00:41:06.163273 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:06.163257 5890 scope.go:122] "RemoveContainer" containerID="0ebd8a6ccebe6eb6443fe36868eceec141883f3c653882c9aab600050d0d4015" Oct 31 00:41:06.164578 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:06.164547 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:06.164683 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:06.164629 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:06.164683 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:06.164673 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:07.016833 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:41:07.016763 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:41:07.136546 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:07.136514 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 00:41:07.175279 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:07.175242 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" event={"ID":"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7","Type":"ContainerStarted","Data":"03be9857d88c69a09cf0ef0ce37288a42580f954e445afd3a1f362fc07c91a9f"} Oct 31 00:41:07.895594 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:07.895561 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:41:07.895814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:07.895726 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:41:07.895814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:07.895776 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 00:41:08.895952 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:08.895917 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:41:08.896356 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:08.895979 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 00:41:09.896308 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:09.896264 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:41:09.896935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:09.896892 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 00:41:10.895566 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:10.895532 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:41:10.895744 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:10.895586 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 00:41:11.127403 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:11.127376 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 00:41:11.896382 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:11.896344 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:41:11.896589 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:11.896415 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 00:41:12.899198 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:12.899167 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:41:12.899580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:12.899560 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:41:12.900318 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:12.900298 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:41:13.145562 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.145529 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 00:41:13.486961 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.486921 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb"] Oct 31 00:41:13.490227 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.490204 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.523832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.523801 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.523979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.523871 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zkr\" (UniqueName: \"kubernetes.io/projected/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-kube-api-access-b7zkr\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.523979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.523935 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.524099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.524030 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.540307 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.540281 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-dd98f0-serving-cert\"" Oct 31 00:41:13.546935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.546907 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb"] Oct 31 00:41:13.624896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.624855 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zkr\" (UniqueName: \"kubernetes.io/projected/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-kube-api-access-b7zkr\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.625086 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.624951 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.625086 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.625011 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.625215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.625120 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.625295 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:41:13.625274 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-serving-cert: secret "maas-api-e2e-worker-w3-dd98f0-serving-cert" not found Oct 31 00:41:13.625376 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:41:13.625363 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls podName:8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8 nodeName:}" failed. No retries permitted until 2026-10-31 00:41:14.125324042 +0000 UTC m=+766.327574474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls") pod "maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" (UID: "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8") : secret "maas-api-e2e-worker-w3-dd98f0-serving-cert" not found Oct 31 00:41:13.625814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.625769 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.627691 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.627667 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:13.683952 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:13.683918 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zkr\" (UniqueName: \"kubernetes.io/projected/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-kube-api-access-b7zkr\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:14.142047 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:14.141029 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:14.150380 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:14.150346 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:14.402540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:14.402457 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:14.798359 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:14.798305 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb"] Oct 31 00:41:14.799960 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:41:14.799921 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd07ff5_76c6_44e8_8c1a_7f76192bb5f8.slice/crio-cc370edbc0b2550381df1850ba2a4130de23d6b01a7848acda36cfbcd1f7db70 WatchSource:0}: Error finding container cc370edbc0b2550381df1850ba2a4130de23d6b01a7848acda36cfbcd1f7db70: Status 404 returned error can't find the container with id cc370edbc0b2550381df1850ba2a4130de23d6b01a7848acda36cfbcd1f7db70 Oct 31 00:41:15.232404 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:15.232370 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" event={"ID":"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8","Type":"ContainerStarted","Data":"cc370edbc0b2550381df1850ba2a4130de23d6b01a7848acda36cfbcd1f7db70"} Oct 31 00:41:16.239622 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:16.239584 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" event={"ID":"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8","Type":"ContainerStarted","Data":"4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143"} Oct 31 00:41:16.240077 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:16.239690 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:16.354625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:16.354557 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" podStartSLOduration=2.4656214 podStartE2EDuration="3.354534381s" totalImagesPullingTime="888.912981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:16.352887799 +0000 UTC m=+768.555138231" watchObservedRunningTime="2026-10-31 00:41:16.354534381 +0000 UTC m=+768.556784817" Oct 31 00:41:22.252430 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:22.252400 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 00:41:29.012923 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.012886 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh"] Oct 31 00:41:29.016496 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.016477 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.069557 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.069524 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-8bec73-serving-cert\"" Oct 31 00:41:29.115451 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.115420 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91f41c9-0702-4750-94d7-c660db68c749-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.115451 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.115458 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7dq\" (UniqueName: \"kubernetes.io/projected/c91f41c9-0702-4750-94d7-c660db68c749-kube-api-access-4d7dq\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.115669 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.115549 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.115669 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.115619 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-metrics-certs\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.118919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.118900 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh"] Oct 31 00:41:29.217270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.217234 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.217462 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.217327 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-metrics-certs\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.217462 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:41:29.217345 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-serving-cert: secret "maas-api-e2e-worker-w0-8bec73-serving-cert" not found Oct 31 00:41:29.217462 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:41:29.217414 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls podName:c91f41c9-0702-4750-94d7-c660db68c749 nodeName:}" failed. No retries permitted until 2026-10-31 00:41:29.717391804 +0000 UTC m=+781.919642217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls") pod "maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" (UID: "c91f41c9-0702-4750-94d7-c660db68c749") : secret "maas-api-e2e-worker-w0-8bec73-serving-cert" not found Oct 31 00:41:29.217642 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.217525 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91f41c9-0702-4750-94d7-c660db68c749-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.217642 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.217564 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7dq\" (UniqueName: \"kubernetes.io/projected/c91f41c9-0702-4750-94d7-c660db68c749-kube-api-access-4d7dq\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.218364 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.218338 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91f41c9-0702-4750-94d7-c660db68c749-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.220380 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.220357 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-metrics-certs\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.276383 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.276268 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7dq\" (UniqueName: \"kubernetes.io/projected/c91f41c9-0702-4750-94d7-c660db68c749-kube-api-access-4d7dq\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.722796 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.722755 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.725225 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.725204 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls\") pod \"maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:29.931977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:29.931745 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:30.334000 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:30.333971 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh"] Oct 31 00:41:30.335913 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:41:30.335888 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91f41c9_0702_4750_94d7_c660db68c749.slice/crio-f72f0275ffe338fce63c612e43f9f35080d9eda9f97b0f34e38a61d8328a8f92 WatchSource:0}: Error finding container f72f0275ffe338fce63c612e43f9f35080d9eda9f97b0f34e38a61d8328a8f92: Status 404 returned error can't find the container with id f72f0275ffe338fce63c612e43f9f35080d9eda9f97b0f34e38a61d8328a8f92 Oct 31 00:41:31.342730 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:31.342691 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" event={"ID":"c91f41c9-0702-4750-94d7-c660db68c749","Type":"ContainerStarted","Data":"f72f0275ffe338fce63c612e43f9f35080d9eda9f97b0f34e38a61d8328a8f92"} Oct 31 00:41:32.351321 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:32.351277 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" event={"ID":"c91f41c9-0702-4750-94d7-c660db68c749","Type":"ContainerStarted","Data":"cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af"} Oct 31 00:41:32.351779 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:32.351409 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:32.470416 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:32.470359 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" podStartSLOduration=3.5275740239999998 podStartE2EDuration="4.470343424s" totalImagesPullingTime="942.7694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:32.469952241 +0000 UTC m=+784.672202675" watchObservedRunningTime="2026-10-31 00:41:32.470343424 +0000 UTC m=+784.672593854" Oct 31 00:41:34.365498 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:34.365412 5890 generic.go:356] "Generic (PLEG): container finished" podID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerID="b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616" exitCode=0 Oct 31 00:41:34.365498 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:34.365489 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" event={"ID":"467d4202-b55f-4b78-8f8c-07184fbb79b1","Type":"ContainerDied","Data":"b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616"} Oct 31 00:41:34.365956 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:34.365900 5890 scope.go:122] "RemoveContainer" containerID="b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616" Oct 31 00:41:34.367149 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:34.367120 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:34.367246 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:34.367184 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:34.367246 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:34.367215 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:35.036054 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:35.036021 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:41:35.379061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:35.378982 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" event={"ID":"467d4202-b55f-4b78-8f8c-07184fbb79b1","Type":"ContainerStarted","Data":"aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57"} Oct 31 00:41:35.432113 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:35.432077 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:41:35.433179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:35.433158 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:41:35.433322 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:35.433200 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 00:41:36.432695 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:36.432662 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:41:36.433134 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:36.432718 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 00:41:37.018243 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:41:37.018198 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:41:37.432512 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:37.432483 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:41:37.432672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:37.432540 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 00:41:38.376849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:38.376821 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 00:41:38.432263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:38.432232 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:41:38.432415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:38.432277 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 00:41:39.432099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:39.431978 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:41:39.432099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:39.432038 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 00:41:40.473462 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:40.473426 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:41:40.474405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:40.474361 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:41:40.475731 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:41:40.475707 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 00:42:07.041117 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:42:07.041072 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:42:21.092349 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.092262 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv"] Oct 31 00:42:21.095479 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.095459 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.169093 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.169059 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-model-cache\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.169280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.169098 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-dshm\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.169280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.169207 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-home\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.169280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.169233 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6mb\" (UniqueName: \"kubernetes.io/projected/8584857a-893b-4637-beb1-b666e919ae56-kube-api-access-kg6mb\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.169387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.169301 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-tmp-dir\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.169387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.169320 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8584857a-893b-4637-beb1-b666e919ae56-tls-certs\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.192967 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.192934 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"default-dockercfg-w97p4\"" Oct 31 00:42:21.193105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.193006 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"kube-root-ca.crt\"" Oct 31 00:42:21.193105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.193008 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"openshift-service-ca.crt\"" Oct 31 00:42:21.193200 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.193153 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"simulator-w3-dd98f0-kserve-self-signed-certs\"" Oct 31 00:42:21.195308 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.195287 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv"] Oct 31 00:42:21.270540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.270500 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-model-cache\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.270739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.270548 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-dshm\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.270739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.270622 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-home\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.270739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.270645 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6mb\" (UniqueName: \"kubernetes.io/projected/8584857a-893b-4637-beb1-b666e919ae56-kube-api-access-kg6mb\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.270739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.270704 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-tmp-dir\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.270739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.270727 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8584857a-893b-4637-beb1-b666e919ae56-tls-certs\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.271081 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.270965 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-model-cache\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.271081 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.271032 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-home\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.271175 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.271110 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-tmp-dir\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.272895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.272874 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-dshm\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.273446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.273427 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8584857a-893b-4637-beb1-b666e919ae56-tls-certs\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.329939 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.329872 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6mb\" (UniqueName: \"kubernetes.io/projected/8584857a-893b-4637-beb1-b666e919ae56-kube-api-access-kg6mb\") pod \"simulator-w3-dd98f0-kserve-c467f8d55-x94mv\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.437440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.437407 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:21.810479 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:21.810452 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv"] Oct 31 00:42:21.811754 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:42:21.811728 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8584857a_893b_4637_beb1_b666e919ae56.slice/crio-b89de8718383103cd2a9460416be53c15a5e63735bbb9dbd5a243b97efd8e0c9 WatchSource:0}: Error finding container b89de8718383103cd2a9460416be53c15a5e63735bbb9dbd5a243b97efd8e0c9: Status 404 returned error can't find the container with id b89de8718383103cd2a9460416be53c15a5e63735bbb9dbd5a243b97efd8e0c9 Oct 31 00:42:22.683417 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:22.683377 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" event={"ID":"8584857a-893b-4637-beb1-b666e919ae56","Type":"ContainerStarted","Data":"4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e"} Oct 31 00:42:22.683417 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:22.683422 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" event={"ID":"8584857a-893b-4637-beb1-b666e919ae56","Type":"ContainerStarted","Data":"b89de8718383103cd2a9460416be53c15a5e63735bbb9dbd5a243b97efd8e0c9"} Oct 31 00:42:22.800254 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:22.800200 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podStartSLOduration=1.800186483 podStartE2EDuration="1.800186483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:22.798390561 +0000 UTC m=+835.000640994" watchObservedRunningTime="2026-10-31 00:42:22.800186483 +0000 UTC m=+835.002436915" Oct 31 00:42:23.438235 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:23.438200 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:31.438212 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:31.438176 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:31.451548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:31.451519 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:31.746294 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:31.746214 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:42:37.018138 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:42:37.018099 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:42:37.087698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.087664 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6"] Oct 31 00:42:37.090841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.090823 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.137021 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.136993 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-tmp-dir\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.137158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.137043 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvnx\" (UniqueName: \"kubernetes.io/projected/bab22885-f796-4636-9e98-7622e5a6757a-kube-api-access-vjvnx\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.137158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.137144 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab22885-f796-4636-9e98-7622e5a6757a-tls-certs\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.137249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.137186 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-dshm\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.137249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.137209 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-model-cache\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.137334 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.137302 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-home\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.191674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.191641 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"kube-root-ca.crt\"" Oct 31 00:42:37.191869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.191681 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"openshift-service-ca.crt\"" Oct 31 00:42:37.191869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.191746 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"default-dockercfg-rm4ww\"" Oct 31 00:42:37.191954 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.191887 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"simulator-w0-8bec73-kserve-self-signed-certs\"" Oct 31 00:42:37.199777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.196688 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6"] Oct 31 00:42:37.238733 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.238698 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-tmp-dir\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.238944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.238764 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvnx\" (UniqueName: \"kubernetes.io/projected/bab22885-f796-4636-9e98-7622e5a6757a-kube-api-access-vjvnx\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.238944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.238849 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab22885-f796-4636-9e98-7622e5a6757a-tls-certs\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.238944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.238891 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-dshm\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.238944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.238914 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-model-cache\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.238944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.238941 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-home\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.239248 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.239200 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-tmp-dir\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.239667 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.239635 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-model-cache\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.239768 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.239662 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-home\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.241524 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.241498 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-dshm\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.242154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.242122 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab22885-f796-4636-9e98-7622e5a6757a-tls-certs\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.297327 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.297252 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvnx\" (UniqueName: \"kubernetes.io/projected/bab22885-f796-4636-9e98-7622e5a6757a-kube-api-access-vjvnx\") pod \"simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.401400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.401355 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:37.771554 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:37.771526 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6"] Oct 31 00:42:37.773474 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:42:37.773448 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbab22885_f796_4636_9e98_7622e5a6757a.slice/crio-824d62c75449be628028b7ba2391dd1e95d6cef8fe5d0aa73d6be69e87843810 WatchSource:0}: Error finding container 824d62c75449be628028b7ba2391dd1e95d6cef8fe5d0aa73d6be69e87843810: Status 404 returned error can't find the container with id 824d62c75449be628028b7ba2391dd1e95d6cef8fe5d0aa73d6be69e87843810 Oct 31 00:42:38.779583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:38.779533 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" event={"ID":"bab22885-f796-4636-9e98-7622e5a6757a","Type":"ContainerStarted","Data":"7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8"} Oct 31 00:42:38.780049 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:38.779622 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" event={"ID":"bab22885-f796-4636-9e98-7622e5a6757a","Type":"ContainerStarted","Data":"824d62c75449be628028b7ba2391dd1e95d6cef8fe5d0aa73d6be69e87843810"} Oct 31 00:42:38.893413 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:38.893357 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podStartSLOduration=1.893343238 podStartE2EDuration="1.893343238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:38.891646922 +0000 UTC m=+851.093897355" watchObservedRunningTime="2026-10-31 00:42:38.893343238 +0000 UTC m=+851.095593671" Oct 31 00:42:39.402112 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:39.402074 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:47.401685 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:47.401652 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:47.414854 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:47.414828 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:47.844766 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:47.844734 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:42:58.376097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.376053 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x"] Oct 31 00:42:58.379852 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.379826 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.431097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.431066 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ef2572-openshift-default-dockercfg-fvb72\"" Oct 31 00:42:58.471630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.471593 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39409f96-13f4-433b-98c8-e3c680e08f45-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.471774 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.471638 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.471774 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.471741 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.471876 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.471779 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.471876 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.471860 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39409f96-13f4-433b-98c8-e3c680e08f45-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.471953 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.471878 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6m2\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-kube-api-access-bt6m2\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.471986 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.471969 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.472023 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.472009 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.472063 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.472037 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.482707 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.482675 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x"] Oct 31 00:42:58.573985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.573939 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574018 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574071 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574119 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39409f96-13f4-433b-98c8-e3c680e08f45-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574149 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574212 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574251 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574299 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39409f96-13f4-433b-98c8-e3c680e08f45-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574325 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6m2\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-kube-api-access-bt6m2\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574620 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574408 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574620 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574451 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.574714 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.574623 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.575263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.575232 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39409f96-13f4-433b-98c8-e3c680e08f45-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.575410 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.575307 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.576738 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.576714 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.576957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.576940 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39409f96-13f4-433b-98c8-e3c680e08f45-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.633425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.633359 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.633425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.633376 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6m2\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-kube-api-access-bt6m2\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:58.692962 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:58.692921 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:42:59.119147 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:59.119119 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x"] Oct 31 00:42:59.120924 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:42:59.120895 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39409f96_13f4_433b_98c8_e3c680e08f45.slice/crio-e30bbf9a8b1669b9fe7400543b63779f9920a2b843719227085f6cae05d93a5d WatchSource:0}: Error finding container e30bbf9a8b1669b9fe7400543b63779f9920a2b843719227085f6cae05d93a5d: Status 404 returned error can't find the container with id e30bbf9a8b1669b9fe7400543b63779f9920a2b843719227085f6cae05d93a5d Oct 31 00:42:59.123083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:59.123050 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:59.123148 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:59.123116 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:59.123188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:59.123146 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:59.912168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:59.912128 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" event={"ID":"39409f96-13f4-433b-98c8-e3c680e08f45","Type":"ContainerStarted","Data":"bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51"} Oct 31 00:42:59.912627 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:42:59.912183 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" event={"ID":"39409f96-13f4-433b-98c8-e3c680e08f45","Type":"ContainerStarted","Data":"e30bbf9a8b1669b9fe7400543b63779f9920a2b843719227085f6cae05d93a5d"} Oct 31 00:43:00.036462 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:00.036408 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" podStartSLOduration=2.03639304 podStartE2EDuration="2.03639304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:00.034658984 +0000 UTC m=+872.236909417" watchObservedRunningTime="2026-10-31 00:43:00.03639304 +0000 UTC m=+872.238643553" Oct 31 00:43:00.693878 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:00.693846 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:43:00.695585 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:00.695562 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:43:00.695701 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:00.695619 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 00:43:01.693898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:01.693854 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:43:01.694337 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:01.693935 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 00:43:02.694381 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:02.694340 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:43:02.694918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:02.694399 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 00:43:03.693345 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:03.693296 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:43:03.693534 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:03.693362 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 00:43:04.693323 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:04.693288 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:43:04.693716 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:04.693345 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 00:43:05.909923 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:05.909878 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:43:05.910659 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:05.910633 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:43:05.911357 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:05.911337 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 00:43:07.025245 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:43:07.025189 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:43:28.040626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:28.040598 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:43:28.045319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:43:28.045285 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:43:37.018833 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:43:37.018777 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:44:07.107600 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:44:07.107552 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:44:37.017977 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:44:37.017941 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:44:37.372731 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.372648 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb"] Oct 31 00:44:37.376842 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.376820 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538070 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538032 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538070 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538071 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538093 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpmv\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-kube-api-access-ptpmv\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538192 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538338 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538291 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538372 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538348 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538392 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538463 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538449 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.538507 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.538496 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.581534 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.581496 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb"] Oct 31 00:44:37.639523 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639442 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639523 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639485 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639523 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639519 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639572 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639591 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639611 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpmv\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-kube-api-access-ptpmv\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639637 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639711 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.639818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.639752 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.640161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.640033 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.640161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.640075 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.640161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.640113 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.640359 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.640203 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.640553 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.640535 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.642581 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.642555 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.642913 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.642887 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.698280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.698249 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.698421 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.698305 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpmv\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-kube-api-access-ptpmv\") pod \"maas-default-gateway-openshift-default-6cc458d887-z2grb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:37.990267 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:37.990234 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:38.414151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.414125 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb"] Oct 31 00:44:38.416552 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:44:38.416524 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdddbf4a_a26a_4d40_93d6_accc27dc93eb.slice/crio-3efa059e130bb28cd746528a921a5aa31afc4fe96964c5918fd2e5a51404fe93 WatchSource:0}: Error finding container 3efa059e130bb28cd746528a921a5aa31afc4fe96964c5918fd2e5a51404fe93: Status 404 returned error can't find the container with id 3efa059e130bb28cd746528a921a5aa31afc4fe96964c5918fd2e5a51404fe93 Oct 31 00:44:38.419083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.419043 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:44:38.419177 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.419128 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:44:38.419217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.419172 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:44:38.551987 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.551954 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" event={"ID":"fdddbf4a-a26a-4d40-93d6-accc27dc93eb","Type":"ContainerStarted","Data":"9d1adc88be2ee495ca6a4abef1666fbd258c4298daec03b55a218ab0c3829b1f"} Oct 31 00:44:38.552169 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.552012 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" event={"ID":"fdddbf4a-a26a-4d40-93d6-accc27dc93eb","Type":"ContainerStarted","Data":"3efa059e130bb28cd746528a921a5aa31afc4fe96964c5918fd2e5a51404fe93"} Oct 31 00:44:38.672184 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.672140 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podStartSLOduration=1.672124832 podStartE2EDuration="1.672124832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:38.670213323 +0000 UTC m=+970.872463757" watchObservedRunningTime="2026-10-31 00:44:38.672124832 +0000 UTC m=+970.874375265" Oct 31 00:44:38.990986 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:38.990884 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:39.992456 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:39.992420 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:44:39.992827 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:39.992483 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:44:40.991035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:40.991000 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:44:40.991243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:40.991055 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:44:41.990893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:41.990834 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:44:41.991425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:41.990958 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:44:42.323725 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.323648 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6"] Oct 31 00:44:42.327575 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.327558 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.388157 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.387948 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6"] Oct 31 00:44:42.426935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.426906 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"openshift-service-ca.crt\"" Oct 31 00:44:42.427114 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.426957 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"kube-root-ca.crt\"" Oct 31 00:44:42.427114 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.427061 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"default-dockercfg-fr2qg\"" Oct 31 00:44:42.427197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.427130 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"e2e-conflict-a-ea66c31d-kserve-self-signed-certs\"" Oct 31 00:44:42.507527 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.507495 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-model-cache\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.507793 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.507547 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tmp-dir\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.507793 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.507686 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-home\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.507793 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.507719 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tls-certs\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.507912 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.507845 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-dshm\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.507912 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.507879 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws9rr\" (UniqueName: \"kubernetes.io/projected/c0d2c9ee-adaf-4298-a098-466a6489dc4c-kube-api-access-ws9rr\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.608594 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.608513 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-dshm\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.608594 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.608560 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws9rr\" (UniqueName: \"kubernetes.io/projected/c0d2c9ee-adaf-4298-a098-466a6489dc4c-kube-api-access-ws9rr\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.608850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.608606 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-model-cache\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.608850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.608639 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tmp-dir\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.608850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.608683 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-home\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.608850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.608706 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tls-certs\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.609144 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.609122 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tmp-dir\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.609211 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.609141 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-home\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.609315 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.609300 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-model-cache\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.610853 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.610834 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-dshm\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.611239 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.611220 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tls-certs\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.686804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.684851 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws9rr\" (UniqueName: \"kubernetes.io/projected/c0d2c9ee-adaf-4298-a098-466a6489dc4c-kube-api-access-ws9rr\") pod \"e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.940515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.940485 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:42.991815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.991361 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:44:42.991815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:42.991421 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:44:43.354868 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:43.354117 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6"] Oct 31 00:44:43.600889 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:43.600843 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" event={"ID":"c0d2c9ee-adaf-4298-a098-466a6489dc4c","Type":"ContainerStarted","Data":"9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354"} Oct 31 00:44:43.601051 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:43.600902 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" event={"ID":"c0d2c9ee-adaf-4298-a098-466a6489dc4c","Type":"ContainerStarted","Data":"bce2017f473fa1d911446e9ca3d00c3e820c2b73c6b69fe19589e39fff3bc93f"} Oct 31 00:44:43.715820 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:43.715731 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" podStartSLOduration=1.7157178690000001 podStartE2EDuration="1.715717869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:43.715145601 +0000 UTC m=+975.917396034" watchObservedRunningTime="2026-10-31 00:44:43.715717869 +0000 UTC m=+975.917968302" Oct 31 00:44:43.941742 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:43.941711 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:43.991561 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:43.991471 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:44:43.991561 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:43.991534 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:44:44.991494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:44.991453 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:44:44.992047 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:44.991521 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:44:45.991646 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:45.991606 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:44:45.992289 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:45.992256 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 00:44:47.010235 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:47.010191 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:47.010830 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:47.010778 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.628809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:47.628747 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:47.629262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:47.629237 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:47.630292 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:47.630267 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 00:44:50.536911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.536872 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt"] Oct 31 00:44:50.540891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.540871 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.635641 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.635594 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlklh\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-kube-api-access-mlklh\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.635851 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.635667 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-plugins-config-volume\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.635851 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.635768 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.635958 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.635911 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-routing-overlay-volume\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.640758 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.640735 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bf4919d00529\"" Oct 31 00:44:50.640908 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.640799 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 00:44:50.640908 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.640822 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 00:44:50.641030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.640963 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zh9bc\"" Oct 31 00:44:50.641030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.640973 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 00:44:50.641030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.641014 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 00:44:50.643486 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.643466 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt"] Oct 31 00:44:50.737125 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.737082 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlklh\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-kube-api-access-mlklh\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.737326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.737138 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-plugins-config-volume\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.737326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.737179 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.737326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.737237 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-routing-overlay-volume\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.737990 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.737964 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-routing-overlay-volume\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.738108 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.738062 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-plugins-config-volume\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.739686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.739667 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.796158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.796090 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlklh\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-kube-api-access-mlklh\") pod \"payload-processing-external-model-58d4f57fc5-wd9lt\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:50.854979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:50.854942 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:51.225722 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:51.225692 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt"] Oct 31 00:44:51.227322 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:44:51.227288 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c5096c_b562_4b78_a5ac_90fbbcd8a377.slice/crio-38a9d5c80c7b54c4e8bcdcf8ea0546038858ac2ccdacaa052038b2606d9f692a WatchSource:0}: Error finding container 38a9d5c80c7b54c4e8bcdcf8ea0546038858ac2ccdacaa052038b2606d9f692a: Status 404 returned error can't find the container with id 38a9d5c80c7b54c4e8bcdcf8ea0546038858ac2ccdacaa052038b2606d9f692a Oct 31 00:44:51.668083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:51.668043 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" event={"ID":"79c5096c-b562-4b78-a5ac-90fbbcd8a377","Type":"ContainerStarted","Data":"ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4"} Oct 31 00:44:51.668513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:51.668111 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" event={"ID":"79c5096c-b562-4b78-a5ac-90fbbcd8a377","Type":"ContainerStarted","Data":"38a9d5c80c7b54c4e8bcdcf8ea0546038858ac2ccdacaa052038b2606d9f692a"} Oct 31 00:44:51.668513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:51.668248 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:44:51.783222 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:51.783107 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podStartSLOduration=1.7830916110000001 podStartE2EDuration="1.783091611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:51.782013839 +0000 UTC m=+983.984264271" watchObservedRunningTime="2026-10-31 00:44:51.783091611 +0000 UTC m=+983.985342067" Oct 31 00:44:52.941359 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:52.941329 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:52.959389 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:52.959356 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:44:53.700372 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:44:53.700343 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:45:00.198798 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.198744 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf"] Oct 31 00:45:00.203954 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.203931 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.256573 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.256542 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:45:00.298385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.298346 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgzf\" (UniqueName: \"kubernetes.io/projected/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-kube-api-access-wwgzf\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.298568 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.298539 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.307234 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.307192 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf"] Oct 31 00:45:00.399239 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.399197 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgzf\" (UniqueName: \"kubernetes.io/projected/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-kube-api-access-wwgzf\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.399417 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.399308 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.399970 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.399951 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.458978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.458901 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgzf\" (UniqueName: \"kubernetes.io/projected/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-kube-api-access-wwgzf\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.518086 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.518051 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:00.888102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:00.888073 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf"] Oct 31 00:45:00.889754 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:45:00.889728 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27681f3f_bef8_4cf7_9f50_7ecfb0787db3.slice/crio-69f4ef309a6017fa61f06c9f91914b7824e43efb19c82c6a37939c640fc54fcb WatchSource:0}: Error finding container 69f4ef309a6017fa61f06c9f91914b7824e43efb19c82c6a37939c640fc54fcb: Status 404 returned error can't find the container with id 69f4ef309a6017fa61f06c9f91914b7824e43efb19c82c6a37939c640fc54fcb Oct 31 00:45:01.784559 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:01.784521 5890 generic.go:356] "Generic (PLEG): container finished" podID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerID="0ac9a87244bb260e64fba6c143f187c052a8d1aa20dd838a1151eb8b87f09abc" exitCode=60 Oct 31 00:45:01.785025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:01.784610 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" event={"ID":"27681f3f-bef8-4cf7-9f50-7ecfb0787db3","Type":"ContainerDied","Data":"0ac9a87244bb260e64fba6c143f187c052a8d1aa20dd838a1151eb8b87f09abc"} Oct 31 00:45:01.785025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:01.784690 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" event={"ID":"27681f3f-bef8-4cf7-9f50-7ecfb0787db3","Type":"ContainerStarted","Data":"69f4ef309a6017fa61f06c9f91914b7824e43efb19c82c6a37939c640fc54fcb"} Oct 31 00:45:01.785025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:01.784901 5890 scope.go:122] "RemoveContainer" containerID="0ac9a87244bb260e64fba6c143f187c052a8d1aa20dd838a1151eb8b87f09abc" Oct 31 00:45:02.684843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:02.684803 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:02.793274 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:02.793241 5890 generic.go:356] "Generic (PLEG): container finished" podID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerID="685dbd8c7877819896f8c07a1527e08e59c173d36ba440ae4e6a3df854bc3421" exitCode=60 Oct 31 00:45:02.793719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:02.793306 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" event={"ID":"27681f3f-bef8-4cf7-9f50-7ecfb0787db3","Type":"ContainerDied","Data":"685dbd8c7877819896f8c07a1527e08e59c173d36ba440ae4e6a3df854bc3421"} Oct 31 00:45:02.793719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:02.793355 5890 scope.go:122] "RemoveContainer" containerID="0ac9a87244bb260e64fba6c143f187c052a8d1aa20dd838a1151eb8b87f09abc" Oct 31 00:45:02.793719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:02.793603 5890 scope.go:122] "RemoveContainer" containerID="685dbd8c7877819896f8c07a1527e08e59c173d36ba440ae4e6a3df854bc3421" Oct 31 00:45:02.793935 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:02.793910 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf_odh-ai-gateway-infra(27681f3f-bef8-4cf7-9f50-7ecfb0787db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" Oct 31 00:45:03.802439 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:03.802401 5890 scope.go:122] "RemoveContainer" containerID="685dbd8c7877819896f8c07a1527e08e59c173d36ba440ae4e6a3df854bc3421" Oct 31 00:45:03.802920 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:03.802682 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf_odh-ai-gateway-infra(27681f3f-bef8-4cf7-9f50-7ecfb0787db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" Oct 31 00:45:07.021473 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:07.021430 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:45:12.682993 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:12.682901 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:14.926031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:14.925992 5890 scope.go:122] "RemoveContainer" containerID="685dbd8c7877819896f8c07a1527e08e59c173d36ba440ae4e6a3df854bc3421" Oct 31 00:45:15.879719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:15.879680 5890 generic.go:356] "Generic (PLEG): container finished" podID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerID="af33851b8728d0f6fa0ae83399ee9b527ac373c1bdde9a60cc6a75088eef706d" exitCode=60 Oct 31 00:45:15.879927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:15.879757 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" event={"ID":"27681f3f-bef8-4cf7-9f50-7ecfb0787db3","Type":"ContainerDied","Data":"af33851b8728d0f6fa0ae83399ee9b527ac373c1bdde9a60cc6a75088eef706d"} Oct 31 00:45:15.879927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:15.879869 5890 scope.go:122] "RemoveContainer" containerID="685dbd8c7877819896f8c07a1527e08e59c173d36ba440ae4e6a3df854bc3421" Oct 31 00:45:15.880272 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:15.880245 5890 scope.go:122] "RemoveContainer" containerID="af33851b8728d0f6fa0ae83399ee9b527ac373c1bdde9a60cc6a75088eef706d" Oct 31 00:45:15.880530 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:15.880504 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf_odh-ai-gateway-infra(27681f3f-bef8-4cf7-9f50-7ecfb0787db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" Oct 31 00:45:17.008686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.008648 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf"] Oct 31 00:45:17.154004 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.153978 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:17.191124 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.191095 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-kube-api-access-wwgzf\" (UniqueName: \"kubernetes.io/projected/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-kube-api-access-wwgzf\") pod \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " Oct 31 00:45:17.191305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.191159 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-maas-api-service-ca\") pod \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\" (UID: \"27681f3f-bef8-4cf7-9f50-7ecfb0787db3\") " Oct 31 00:45:17.191542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.191516 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-maas-api-service-ca" pod "27681f3f-bef8-4cf7-9f50-7ecfb0787db3" (UID: "27681f3f-bef8-4cf7-9f50-7ecfb0787db3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:17.191613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.191600 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:17.193394 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.193359 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-kube-api-access-wwgzf" pod "27681f3f-bef8-4cf7-9f50-7ecfb0787db3" (UID: "27681f3f-bef8-4cf7-9f50-7ecfb0787db3"). InnerVolumeSpecName "kube-api-access-wwgzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:17.293091 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.293009 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwgzf\" (UniqueName: \"kubernetes.io/projected/27681f3f-bef8-4cf7-9f50-7ecfb0787db3-kube-api-access-wwgzf\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:17.899681 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.899651 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" Oct 31 00:45:17.899681 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.899663 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf" event={"ID":"27681f3f-bef8-4cf7-9f50-7ecfb0787db3","Type":"ContainerDied","Data":"69f4ef309a6017fa61f06c9f91914b7824e43efb19c82c6a37939c640fc54fcb"} Oct 31 00:45:17.899967 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:17.899728 5890 scope.go:122] "RemoveContainer" containerID="af33851b8728d0f6fa0ae83399ee9b527ac373c1bdde9a60cc6a75088eef706d" Oct 31 00:45:18.065493 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:18.065446 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf"] Oct 31 00:45:18.070352 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:18.070307 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890125-jxnmf"] Oct 31 00:45:19.934998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:19.934953 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" path="/var/lib/kubelet/pods/27681f3f-bef8-4cf7-9f50-7ecfb0787db3/volumes" Oct 31 00:45:22.683390 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:22.683351 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:24.572764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.572727 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf"] Oct 31 00:45:24.573278 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.573262 5890 cpu_manager.go:414] "containerMap: removing container" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:24.573323 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.573280 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:24.573323 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.573299 5890 cpu_manager.go:414] "containerMap: removing container" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:24.573323 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.573306 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:24.573420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.573387 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:24.573420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.573395 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:24.573420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.573404 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:24.578109 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.578092 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.671591 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.671557 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4093e5df-ad6a-467e-a54a-c266fd6df904-tls-certs\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.671777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.671603 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-tmp-dir\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.671777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.671661 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-home\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.671777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.671735 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-dshm\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.671998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.671810 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4m7j\" (UniqueName: \"kubernetes.io/projected/4093e5df-ad6a-467e-a54a-c266fd6df904-kube-api-access-z4m7j\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.671998 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.671874 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-model-cache\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.676630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.676610 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"distinct-w3-dd98f0-kserve-self-signed-certs\"" Oct 31 00:45:24.679131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.679108 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf"] Oct 31 00:45:24.773313 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773275 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-home\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.773572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773405 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-dshm\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.773572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773435 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4m7j\" (UniqueName: \"kubernetes.io/projected/4093e5df-ad6a-467e-a54a-c266fd6df904-kube-api-access-z4m7j\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.773572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773478 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-model-cache\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.773755 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773590 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4093e5df-ad6a-467e-a54a-c266fd6df904-tls-certs\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.773755 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773637 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-tmp-dir\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.773894 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773816 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-home\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.773950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.773891 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-model-cache\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.774084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.774044 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-tmp-dir\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.775797 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.775767 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-dshm\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.776490 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.776467 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4093e5df-ad6a-467e-a54a-c266fd6df904-tls-certs\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.834986 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.834904 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4m7j\" (UniqueName: \"kubernetes.io/projected/4093e5df-ad6a-467e-a54a-c266fd6df904-kube-api-access-z4m7j\") pod \"distinct-w3-dd98f0-kserve-845cf54849-b29nf\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:24.890504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:24.890470 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:25.269446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:25.269417 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf"] Oct 31 00:45:25.270554 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:45:25.270529 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4093e5df_ad6a_467e_a54a_c266fd6df904.slice/crio-ffa53682ab843fde2843ddf4eda0a10fa69865795bb5163f86698218772b77a0 WatchSource:0}: Error finding container ffa53682ab843fde2843ddf4eda0a10fa69865795bb5163f86698218772b77a0: Status 404 returned error can't find the container with id ffa53682ab843fde2843ddf4eda0a10fa69865795bb5163f86698218772b77a0 Oct 31 00:45:25.957046 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:25.957007 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" event={"ID":"4093e5df-ad6a-467e-a54a-c266fd6df904","Type":"ContainerStarted","Data":"65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d"} Oct 31 00:45:25.957426 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:25.957056 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" event={"ID":"4093e5df-ad6a-467e-a54a-c266fd6df904","Type":"ContainerStarted","Data":"ffa53682ab843fde2843ddf4eda0a10fa69865795bb5163f86698218772b77a0"} Oct 31 00:45:26.074152 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:26.074098 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podStartSLOduration=2.074084087 podStartE2EDuration="2.074084087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:26.071046436 +0000 UTC m=+1018.273296869" watchObservedRunningTime="2026-10-31 00:45:26.074084087 +0000 UTC m=+1018.276334570" Oct 31 00:45:26.891275 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:26.891243 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:27.719896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.719863 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45"] Oct 31 00:45:27.720372 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.720356 5890 cpu_manager.go:414] "containerMap: removing container" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:27.720372 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.720369 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27681f3f-bef8-4cf7-9f50-7ecfb0787db3" containerName="cleanup" Oct 31 00:45:27.723664 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.723647 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.779112 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.779077 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45"] Oct 31 00:45:27.810211 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.810179 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-tmp-dir\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.810379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.810226 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-home\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.810379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.810252 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-model-cache\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.810468 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.810400 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f176e3c-5114-48dd-880f-2150d4bc5b22-tls-certs\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.810468 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.810432 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2ll\" (UniqueName: \"kubernetes.io/projected/2f176e3c-5114-48dd-880f-2150d4bc5b22-kube-api-access-5t2ll\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.810578 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.810549 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-dshm\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.824074 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.824023 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"distinct-w0-8bec73-kserve-self-signed-certs\"" Oct 31 00:45:27.911347 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911317 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-tmp-dir\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.911536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911362 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-home\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.911536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911385 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-model-cache\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.911536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911438 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f176e3c-5114-48dd-880f-2150d4bc5b22-tls-certs\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.911536 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911463 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2ll\" (UniqueName: \"kubernetes.io/projected/2f176e3c-5114-48dd-880f-2150d4bc5b22-kube-api-access-5t2ll\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.911700 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911548 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-dshm\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.911907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911885 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-home\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.911983 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.911920 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-model-cache\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.912079 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.912062 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-tmp-dir\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.913855 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.913836 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-dshm\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.960780 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.960755 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"distinct-w0-8bec73-kserve-self-signed-certs\"" Oct 31 00:45:27.963972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.963944 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f176e3c-5114-48dd-880f-2150d4bc5b22-tls-certs\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:27.973078 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:27.973007 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2ll\" (UniqueName: \"kubernetes.io/projected/2f176e3c-5114-48dd-880f-2150d4bc5b22-kube-api-access-5t2ll\") pod \"distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:28.036876 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:28.036839 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:28.418377 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:28.418201 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45"] Oct 31 00:45:28.420915 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:45:28.420884 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f176e3c_5114_48dd_880f_2150d4bc5b22.slice/crio-261e8ea6c98fffe60b1a9389d4d7f9272078ffd3269fdfc2ab00d3624694df9c WatchSource:0}: Error finding container 261e8ea6c98fffe60b1a9389d4d7f9272078ffd3269fdfc2ab00d3624694df9c: Status 404 returned error can't find the container with id 261e8ea6c98fffe60b1a9389d4d7f9272078ffd3269fdfc2ab00d3624694df9c Oct 31 00:45:28.976630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:28.976588 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" event={"ID":"2f176e3c-5114-48dd-880f-2150d4bc5b22","Type":"ContainerStarted","Data":"46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e"} Oct 31 00:45:28.977124 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:28.976641 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" event={"ID":"2f176e3c-5114-48dd-880f-2150d4bc5b22","Type":"ContainerStarted","Data":"261e8ea6c98fffe60b1a9389d4d7f9272078ffd3269fdfc2ab00d3624694df9c"} Oct 31 00:45:29.037904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:29.037872 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:29.091087 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:29.091034 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podStartSLOduration=2.091020674 podStartE2EDuration="2.091020674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:29.089306252 +0000 UTC m=+1021.291556687" watchObservedRunningTime="2026-10-31 00:45:29.091020674 +0000 UTC m=+1021.293271104" Oct 31 00:45:32.682954 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:32.682915 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:34.891314 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:34.891277 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:34.905444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:34.905412 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:35.033300 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:35.033266 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:45:37.018194 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:37.018154 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:45:38.037286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:38.037246 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:38.050181 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:38.050151 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:39.048979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:39.048945 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:45:42.682728 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:42.682691 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:49.473255 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.473209 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns"] Oct 31 00:45:49.477994 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.477973 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.553325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.553289 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns"] Oct 31 00:45:49.557076 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.557049 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-dshm\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.557219 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.557107 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-model-cache\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.557219 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.557192 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tls-certs\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.557336 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.557224 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tmp-dir\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.557393 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.557350 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/4ddbafcb-dbbb-48cf-8145-11628800ff7b-kube-api-access-jbx4z\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.557393 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.557384 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-home\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.579907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.579877 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"e2e-conflict-b-ea66c31d-kserve-self-signed-certs\"" Oct 31 00:45:49.658386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.658348 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tls-certs\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.658592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.658395 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tmp-dir\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.658592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.658467 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/4ddbafcb-dbbb-48cf-8145-11628800ff7b-kube-api-access-jbx4z\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.658592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.658503 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-home\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.658592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.658576 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-dshm\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.658908 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.658636 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-model-cache\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.659000 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.658956 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-home\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.659062 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.659005 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tmp-dir\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.659062 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.659044 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-model-cache\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.661470 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.661447 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-dshm\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.661778 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.661758 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tls-certs\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.717364 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.717333 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/4ddbafcb-dbbb-48cf-8145-11628800ff7b-kube-api-access-jbx4z\") pod \"e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:49.792017 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:49.791915 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:50.197286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:50.197260 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns"] Oct 31 00:45:50.198646 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:45:50.198613 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ddbafcb_dbbb_48cf_8145_11628800ff7b.slice/crio-d85b9645018588b32f83d460f61a17e6921f44a197c3a5075412b674ffafde38 WatchSource:0}: Error finding container d85b9645018588b32f83d460f61a17e6921f44a197c3a5075412b674ffafde38: Status 404 returned error can't find the container with id d85b9645018588b32f83d460f61a17e6921f44a197c3a5075412b674ffafde38 Oct 31 00:45:51.121708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:51.121668 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" event={"ID":"4ddbafcb-dbbb-48cf-8145-11628800ff7b","Type":"ContainerStarted","Data":"e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6"} Oct 31 00:45:51.121708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:51.121712 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" event={"ID":"4ddbafcb-dbbb-48cf-8145-11628800ff7b","Type":"ContainerStarted","Data":"d85b9645018588b32f83d460f61a17e6921f44a197c3a5075412b674ffafde38"} Oct 31 00:45:51.238489 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:51.238435 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" podStartSLOduration=2.238421149 podStartE2EDuration="2.238421149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:51.234954847 +0000 UTC m=+1043.437205282" watchObservedRunningTime="2026-10-31 00:45:51.238421149 +0000 UTC m=+1043.440671620" Oct 31 00:45:51.792094 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:51.792004 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:52.683460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:52.683425 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:54.927089 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:54.927048 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:55.027208 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:55.027174 5890 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4919d00529: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:55.027453 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:55.027428 5890 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:55.027546 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:55.027516 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials podName:79c5096c-b562-4b78-a5ac-90fbbcd8a377 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:55.527494638 +0000 UTC m=+1047.729745074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials") pod "payload-processing-external-model-58d4f57fc5-wd9lt" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377") : secret "agc-cred-bf4919d00529" not found Oct 31 00:45:55.534409 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:55.534361 5890 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4919d00529: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:55.534409 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:55.534399 5890 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:55.534675 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:55.534460 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials podName:79c5096c-b562-4b78-a5ac-90fbbcd8a377 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:56.534443045 +0000 UTC m=+1048.736693469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials") pod "payload-processing-external-model-58d4f57fc5-wd9lt" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377") : secret "agc-cred-bf4919d00529" not found Oct 31 00:45:56.545516 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:56.545484 5890 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4919d00529: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:56.545516 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:56.545511 5890 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:56.546101 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:56.545573 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials podName:79c5096c-b562-4b78-a5ac-90fbbcd8a377 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:58.545558187 +0000 UTC m=+1050.747808599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials") pod "payload-processing-external-model-58d4f57fc5-wd9lt" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377") : secret "agc-cred-bf4919d00529" not found Oct 31 00:45:57.220247 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:57.220214 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt"] Oct 31 00:45:57.221487 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:57.221457 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:45:58.570304 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:58.570267 5890 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4919d00529: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:58.570304 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:58.570297 5890 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt: secret "agc-cred-bf4919d00529" not found Oct 31 00:45:58.570891 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:45:58.570355 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials podName:79c5096c-b562-4b78-a5ac-90fbbcd8a377 nodeName:}" failed. No retries permitted until 2026-10-31 00:46:02.570337081 +0000 UTC m=+1054.772587508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials") pod "payload-processing-external-model-58d4f57fc5-wd9lt" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377") : secret "agc-cred-bf4919d00529" not found Oct 31 00:45:59.792053 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:59.792014 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:45:59.805567 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:45:59.805537 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:46:00.194984 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:00.194939 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:46:02.221031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.220977 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" containerID="cri-o://ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4" gracePeriod=25 Oct 31 00:46:02.574595 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.574200 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:46:02.624753 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:46:02.624707 5890 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4919d00529: secret "agc-cred-bf4919d00529" not found Oct 31 00:46:02.624753 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:46:02.624745 5890 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt: secret "agc-cred-bf4919d00529" not found Oct 31 00:46:02.625030 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:46:02.624839 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials podName:79c5096c-b562-4b78-a5ac-90fbbcd8a377 nodeName:}" failed. No retries permitted until 2026-10-31 00:46:10.624817039 +0000 UTC m=+1062.827067471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials") pod "payload-processing-external-model-58d4f57fc5-wd9lt" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377") : secret "agc-cred-bf4919d00529" not found Oct 31 00:46:02.725211 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.725172 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials\") pod \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " Oct 31 00:46:02.725420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.725236 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-kube-api-access-mlklh\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-kube-api-access-mlklh\") pod \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " Oct 31 00:46:02.725420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.725295 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-plugins-config-volume\") pod \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " Oct 31 00:46:02.725586 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.725454 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-routing-overlay-volume\") pod \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\" (UID: \"79c5096c-b562-4b78-a5ac-90fbbcd8a377\") " Oct 31 00:46:02.726282 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.726250 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-routing-overlay-volume" pod "79c5096c-b562-4b78-a5ac-90fbbcd8a377" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:02.727128 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.727085 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-plugins-config-volume" pod "79c5096c-b562-4b78-a5ac-90fbbcd8a377" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:02.730070 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.730030 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-kube-api-access-mlklh" pod "79c5096c-b562-4b78-a5ac-90fbbcd8a377" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377"). InnerVolumeSpecName "kube-api-access-mlklh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:02.730778 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.730756 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials" pod "79c5096c-b562-4b78-a5ac-90fbbcd8a377" (UID: "79c5096c-b562-4b78-a5ac-90fbbcd8a377"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:02.826872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.826727 5890 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-provider-credentials\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:02.826872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.826772 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlklh\" (UniqueName: \"kubernetes.io/projected/79c5096c-b562-4b78-a5ac-90fbbcd8a377-kube-api-access-mlklh\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:02.826872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.826811 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:02.826872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.826826 5890 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79c5096c-b562-4b78-a5ac-90fbbcd8a377-routing-overlay-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:02.830762 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.830731 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp"] Oct 31 00:46:02.831409 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.831390 5890 cpu_manager.go:414] "containerMap: removing container" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" Oct 31 00:46:02.831565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.831410 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" Oct 31 00:46:02.831565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.831545 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerName="payload-processing" Oct 31 00:46:02.835259 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.835235 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.888231 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.888196 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-8bec73-openshift-default-dockercfg-xqgfq\"" Oct 31 00:46:02.935421 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.935382 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.935708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.935686 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.935905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.935885 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5n9\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-kube-api-access-wn5n9\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.936083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.936065 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.936203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.936186 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.936343 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.936327 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.936478 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.936462 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.936609 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.936594 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac77a6db-df07-4cd5-a205-9858cf0be656-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.936724 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.936709 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:02.937390 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:02.937369 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp"] Oct 31 00:46:03.088008 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.087798 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.088285 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.088247 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.090026 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.090002 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.090251 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.090218 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.096939 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.096896 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac77a6db-df07-4cd5-a205-9858cf0be656-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.097241 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.097207 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.097446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.097425 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.097672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.097652 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.097957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.097937 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5n9\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-kube-api-access-wn5n9\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.098580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.090657 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.114882 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.105365 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac77a6db-df07-4cd5-a205-9858cf0be656-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.114882 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.105694 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.114882 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.089551 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.114882 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.106331 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.172821 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.169390 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.207813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.188414 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.217645 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.217564 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5n9\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-kube-api-access-wn5n9\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.226809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.218310 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.280685 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.280645 5890 generic.go:356] "Generic (PLEG): container finished" podID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" containerID="ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4" exitCode=0 Oct 31 00:46:03.293819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.280888 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" Oct 31 00:46:03.293819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.280910 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" event={"ID":"79c5096c-b562-4b78-a5ac-90fbbcd8a377","Type":"ContainerDied","Data":"ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4"} Oct 31 00:46:03.293819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.281936 5890 scope.go:122] "RemoveContainer" containerID="ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4" Oct 31 00:46:03.312151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.312046 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt" event={"ID":"79c5096c-b562-4b78-a5ac-90fbbcd8a377","Type":"ContainerDied","Data":"38a9d5c80c7b54c4e8bcdcf8ea0546038858ac2ccdacaa052038b2606d9f692a"} Oct 31 00:46:03.339523 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.339431 5890 scope.go:122] "RemoveContainer" containerID="ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4" Oct 31 00:46:03.339895 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:46:03.339875 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4\": container with ID starting with ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4 not found: ID does not exist" containerID="ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4" Oct 31 00:46:03.340018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.340002 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4"} err="failed to get container status \"ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4\": rpc error: code = NotFound desc = could not find container \"ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4\": container with ID starting with ee260beac684474865b419efb992d3472965558057f6f2170d17c737d6589ce4 not found: ID does not exist" Oct 31 00:46:03.443611 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.443569 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt"] Oct 31 00:46:03.462808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.455051 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d4f57fc5-wd9lt"] Oct 31 00:46:03.490852 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.474042 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:03.941621 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:03.941539 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c5096c-b562-4b78-a5ac-90fbbcd8a377" path="/var/lib/kubelet/pods/79c5096c-b562-4b78-a5ac-90fbbcd8a377/volumes" Oct 31 00:46:04.039661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.039626 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp"] Oct 31 00:46:04.042459 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:46:04.042418 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac77a6db_df07_4cd5_a205_9858cf0be656.slice/crio-a1312e478a727c2cfe8fb314f41d0b9f097efe1ab37aea2a564f83c46b2cfd74 WatchSource:0}: Error finding container a1312e478a727c2cfe8fb314f41d0b9f097efe1ab37aea2a564f83c46b2cfd74: Status 404 returned error can't find the container with id a1312e478a727c2cfe8fb314f41d0b9f097efe1ab37aea2a564f83c46b2cfd74 Oct 31 00:46:04.053708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.053667 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:04.061957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.061913 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:04.062193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.062167 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:04.342828 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.342558 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" event={"ID":"ac77a6db-df07-4cd5-a205-9858cf0be656","Type":"ContainerStarted","Data":"4d5388655222875c5f0c66350ea3473c4b5c838c8fbdca1933c4a0638e170c92"} Oct 31 00:46:04.342828 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.342619 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" event={"ID":"ac77a6db-df07-4cd5-a205-9858cf0be656","Type":"ContainerStarted","Data":"a1312e478a727c2cfe8fb314f41d0b9f097efe1ab37aea2a564f83c46b2cfd74"} Oct 31 00:46:04.477733 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.477704 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:04.518183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:04.518054 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podStartSLOduration=2.518031139 podStartE2EDuration="2.518031139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:04.482229636 +0000 UTC m=+1056.684480070" watchObservedRunningTime="2026-10-31 00:46:04.518031139 +0000 UTC m=+1056.720281573" Oct 31 00:46:05.477091 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.477052 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:46:05.477488 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.477108 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:46:05.716594 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.716556 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv"] Oct 31 00:46:05.720610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.720593 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.822644 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.822607 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv"] Oct 31 00:46:05.840332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840299 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840335 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840562 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840490 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80500604-236e-4c6c-8ffc-51add6824497-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840614 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840584 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80500604-236e-4c6c-8ffc-51add6824497-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840667 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840612 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840735 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840715 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpn5\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-kube-api-access-6lpn5\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840770 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840873 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840859 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.840909 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.840898 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.941735 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.941701 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.941735 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.941736 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.941979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.941854 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80500604-236e-4c6c-8ffc-51add6824497-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.941979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.941905 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80500604-236e-4c6c-8ffc-51add6824497-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942113 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942030 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942172 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942121 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpn5\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-kube-api-access-6lpn5\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942227 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942170 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942227 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942194 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942227 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942211 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942378 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942293 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942646 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942624 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942828 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942670 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.942948 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942706 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.943052 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.942951 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80500604-236e-4c6c-8ffc-51add6824497-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.964193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.964114 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:05.977868 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:05.977767 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80500604-236e-4c6c-8ffc-51add6824497-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:06.000805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.000760 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:06.001097 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.001069 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpn5\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-kube-api-access-6lpn5\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:06.033914 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.033880 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:06.475195 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.475157 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:46:06.475382 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.475218 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:46:06.491843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.491816 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv"] Oct 31 00:46:06.492582 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:46:06.492558 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80500604_236e_4c6c_8ffc_51add6824497.slice/crio-4a42493b47c5099a99f40a0c0c2e71a219b2e638ae36411954326a202634c5ac WatchSource:0}: Error finding container 4a42493b47c5099a99f40a0c0c2e71a219b2e638ae36411954326a202634c5ac: Status 404 returned error can't find the container with id 4a42493b47c5099a99f40a0c0c2e71a219b2e638ae36411954326a202634c5ac Oct 31 00:46:06.494903 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.494872 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:06.494969 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.494936 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:06.494969 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:06.494965 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:07.019320 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:46:07.019278 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:46:07.404347 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:07.404245 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" event={"ID":"80500604-236e-4c6c-8ffc-51add6824497","Type":"ContainerStarted","Data":"2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39"} Oct 31 00:46:07.404588 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:07.404558 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" event={"ID":"80500604-236e-4c6c-8ffc-51add6824497","Type":"ContainerStarted","Data":"4a42493b47c5099a99f40a0c0c2e71a219b2e638ae36411954326a202634c5ac"} Oct 31 00:46:07.476809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:07.476204 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:46:07.476809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:07.476268 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:46:07.578811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:07.577724 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podStartSLOduration=2.577703913 podStartE2EDuration="2.577703913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:07.57225924 +0000 UTC m=+1059.774509679" watchObservedRunningTime="2026-10-31 00:46:07.577703913 +0000 UTC m=+1059.779954347" Oct 31 00:46:08.034984 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:08.034908 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:08.036486 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:08.036461 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:46:08.036569 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:08.036509 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:46:08.491810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:08.491283 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:46:08.491810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:08.491370 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:46:09.034748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:09.034711 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:46:09.035280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:09.034763 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:46:09.481233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:09.481192 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:46:09.481522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:09.481492 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:46:10.034752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:10.034719 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:46:10.034950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:10.034766 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:46:10.478533 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:10.478370 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:46:10.479128 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:10.479080 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 00:46:11.034396 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:11.034362 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:46:11.034614 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:11.034422 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:46:11.737704 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:11.737619 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:11.738657 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:11.738631 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:11.740142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:11.740118 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 00:46:12.034298 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:12.034211 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:46:12.034298 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:12.034278 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:46:13.035016 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:13.034988 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:46:13.035396 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:13.035035 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 00:46:14.039510 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:14.039471 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:14.039922 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:14.039726 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:14.040383 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:14.040366 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 00:46:14.937597 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:14.937563 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg"] Oct 31 00:46:14.941256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:14.941237 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.039521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.039491 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg"] Oct 31 00:46:15.046259 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046227 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046413 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046275 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2z4v\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-kube-api-access-l2z4v\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046457 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046414 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046497 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046464 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/910c0d85-5973-47b2-84dd-08fb08117773-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046519 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046586 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046569 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046597 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/910c0d85-5973-47b2-84dd-08fb08117773-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046620 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.046711 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.046682 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.147640 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147603 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.147833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147652 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/910c0d85-5973-47b2-84dd-08fb08117773-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.147833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147682 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.147833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147714 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.147833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147731 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/910c0d85-5973-47b2-84dd-08fb08117773-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.147833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147751 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.147833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147778 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.148171 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147905 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.148171 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.147938 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2z4v\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-kube-api-access-l2z4v\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.148171 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.148135 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.148349 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.148168 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.148349 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.148321 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.148452 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.148387 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.148588 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.148567 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/910c0d85-5973-47b2-84dd-08fb08117773-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.150258 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.150238 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.150347 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.150330 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/910c0d85-5973-47b2-84dd-08fb08117773-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.206253 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.206189 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.206384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.206352 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2z4v\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-kube-api-access-l2z4v\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.255347 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.255313 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:15.676521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.676490 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg"] Oct 31 00:46:15.677358 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:46:15.677331 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910c0d85_5973_47b2_84dd_08fb08117773.slice/crio-6149ed5b54646b8f4db104de5c443f989443be481e60fcbaa471868b11b07555 WatchSource:0}: Error finding container 6149ed5b54646b8f4db104de5c443f989443be481e60fcbaa471868b11b07555: Status 404 returned error can't find the container with id 6149ed5b54646b8f4db104de5c443f989443be481e60fcbaa471868b11b07555 Oct 31 00:46:15.679732 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.679698 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:15.679827 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.679771 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:15.679882 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:15.679843 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:16.481748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:16.481709 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" event={"ID":"910c0d85-5973-47b2-84dd-08fb08117773","Type":"ContainerStarted","Data":"9ddd88f218e98276e2789922f04eacc4063e87be26c3969ce0f94b91893210b6"} Oct 31 00:46:16.482229 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:16.481768 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" event={"ID":"910c0d85-5973-47b2-84dd-08fb08117773","Type":"ContainerStarted","Data":"6149ed5b54646b8f4db104de5c443f989443be481e60fcbaa471868b11b07555"} Oct 31 00:46:16.606734 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:16.606665 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podStartSLOduration=2.6066466029999997 podStartE2EDuration="2.606646603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:16.6046537 +0000 UTC m=+1068.806904134" watchObservedRunningTime="2026-10-31 00:46:16.606646603 +0000 UTC m=+1068.808897039" Oct 31 00:46:17.259162 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:17.259126 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:17.271810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:17.269926 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:46:17.271810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:17.269991 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:46:18.259810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:18.256184 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:46:18.259810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:18.256248 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:46:19.257227 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:19.257186 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:46:19.257548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:19.257516 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:46:20.257209 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:20.257171 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:46:20.257890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:20.257838 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:46:21.259647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:21.259171 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:46:21.259647 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:21.259231 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:46:22.256004 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:22.255969 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:46:22.256178 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:22.256043 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 00:46:23.260251 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:23.260219 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:23.260661 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:23.260639 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:23.261284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:23.261263 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:46:25.725596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.725558 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rhclr-8fxxd"] Oct 31 00:46:25.731556 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.731536 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.781560 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.781529 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rhclr-8fxxd"] Oct 31 00:46:25.790691 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.790665 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2e885952-4fd9-452a-a737-e5dae9e412a5-archives-path\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.790848 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.790835 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88svn\" (UniqueName: \"kubernetes.io/projected/2e885952-4fd9-452a-a737-e5dae9e412a5-kube-api-access-88svn\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.790934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.790877 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-service-ca-bundle\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.790934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.790892 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-trusted-ca-bundle\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.892070 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.892038 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2e885952-4fd9-452a-a737-e5dae9e412a5-archives-path\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.892257 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.892197 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88svn\" (UniqueName: \"kubernetes.io/projected/2e885952-4fd9-452a-a737-e5dae9e412a5-kube-api-access-88svn\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.892329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.892257 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-service-ca-bundle\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.892329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.892282 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-trusted-ca-bundle\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.892560 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.892537 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2e885952-4fd9-452a-a737-e5dae9e412a5-archives-path\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.893205 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.893176 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-service-ca-bundle\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.893362 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.893341 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-trusted-ca-bundle\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:25.951242 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:25.951208 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88svn\" (UniqueName: \"kubernetes.io/projected/2e885952-4fd9-452a-a737-e5dae9e412a5-kube-api-access-88svn\") pod \"periodic-gathering-rhclr-8fxxd\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:26.045162 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:26.045086 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:46:26.432225 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:26.432195 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rhclr-8fxxd"] Oct 31 00:46:26.433754 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:46:26.433719 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e885952_4fd9_452a_a737_e5dae9e412a5.slice/crio-1f0fcafdb3b5bec2f0fe79ceb836c955ee56e18a76887333f509ac503dd6048a WatchSource:0}: Error finding container 1f0fcafdb3b5bec2f0fe79ceb836c955ee56e18a76887333f509ac503dd6048a: Status 404 returned error can't find the container with id 1f0fcafdb3b5bec2f0fe79ceb836c955ee56e18a76887333f509ac503dd6048a Oct 31 00:46:26.552041 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:26.552006 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" event={"ID":"2e885952-4fd9-452a-a737-e5dae9e412a5","Type":"ContainerStarted","Data":"0f8262a9417b2643c1f80d6bd7d8d9cdfc0acd4e463b2272405b1eaab2bc286b"} Oct 31 00:46:26.552296 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:26.552063 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" event={"ID":"2e885952-4fd9-452a-a737-e5dae9e412a5","Type":"ContainerStarted","Data":"1f0fcafdb3b5bec2f0fe79ceb836c955ee56e18a76887333f509ac503dd6048a"} Oct 31 00:46:26.665937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:26.665885 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" podStartSLOduration=1.665872349 podStartE2EDuration="1.665872349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:26.664884092 +0000 UTC m=+1078.867134692" watchObservedRunningTime="2026-10-31 00:46:26.665872349 +0000 UTC m=+1078.868122782" Oct 31 00:46:28.172108 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:46:28.172078 5890 scope.go:122] "RemoveContainer" containerID="65e46998b186e904051db49809ae5288e4e96470c5a4dd4be91850855da5233b" Oct 31 00:46:37.018228 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:46:37.018178 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:47:07.018853 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:47:07.018811 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:47:16.393428 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.393389 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5"] Oct 31 00:47:16.397270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.397250 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.434624 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.434593 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1e960f-622b-4653-9725-962d26889964-tls-certs\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.434812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.434636 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-tmp-dir\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.434812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.434677 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-home\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.434945 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.434837 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-dshm\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.434945 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.434905 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgj6p\" (UniqueName: \"kubernetes.io/projected/8e1e960f-622b-4653-9725-962d26889964-kube-api-access-kgj6p\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.435063 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.434942 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-model-cache\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.452416 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.452386 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5"] Oct 31 00:47:16.496437 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.496408 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"distinct-2-w3-dd98f0-kserve-self-signed-certs\"" Oct 31 00:47:16.536670 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.536636 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgj6p\" (UniqueName: \"kubernetes.io/projected/8e1e960f-622b-4653-9725-962d26889964-kube-api-access-kgj6p\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.536840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.536682 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-model-cache\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.536840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.536767 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1e960f-622b-4653-9725-962d26889964-tls-certs\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.536840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.536826 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-tmp-dir\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.536966 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.536860 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-home\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.536966 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.536918 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-dshm\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.537223 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.537201 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-model-cache\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.537271 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.537252 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-tmp-dir\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.537310 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.537279 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-home\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.539203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.539168 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-dshm\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.539463 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.539446 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1e960f-622b-4653-9725-962d26889964-tls-certs\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.595742 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.595712 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgj6p\" (UniqueName: \"kubernetes.io/projected/8e1e960f-622b-4653-9725-962d26889964-kube-api-access-kgj6p\") pod \"distinct-2-w3-dd98f0-kserve-754859678b-dtcq5\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:16.709252 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:16.709222 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:17.127916 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:17.127886 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5"] Oct 31 00:47:17.129855 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:47:17.129819 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e1e960f_622b_4653_9725_962d26889964.slice/crio-d8a419d11e20829ac1e2355ba6787d294c81c79434dafc6581c19b0286708b0d WatchSource:0}: Error finding container d8a419d11e20829ac1e2355ba6787d294c81c79434dafc6581c19b0286708b0d: Status 404 returned error can't find the container with id d8a419d11e20829ac1e2355ba6787d294c81c79434dafc6581c19b0286708b0d Oct 31 00:47:17.903834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:17.903758 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" event={"ID":"8e1e960f-622b-4653-9725-962d26889964","Type":"ContainerStarted","Data":"a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe"} Oct 31 00:47:17.904330 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:17.903846 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" event={"ID":"8e1e960f-622b-4653-9725-962d26889964","Type":"ContainerStarted","Data":"d8a419d11e20829ac1e2355ba6787d294c81c79434dafc6581c19b0286708b0d"} Oct 31 00:47:18.019820 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:18.019730 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podStartSLOduration=2.019715936 podStartE2EDuration="2.019715936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:18.016641746 +0000 UTC m=+1130.218892177" watchObservedRunningTime="2026-10-31 00:47:18.019715936 +0000 UTC m=+1130.221966369" Oct 31 00:47:18.611985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:18.611948 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns"] Oct 31 00:47:18.710665 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:18.709934 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:24.455382 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:24.455348 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6"] Oct 31 00:47:26.709777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:26.709745 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:26.722733 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:26.722708 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:26.998099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:26.998023 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:47:28.217808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:28.214329 5890 scope.go:122] "RemoveContainer" containerID="48f68046442dd9da0498437bc1ea97365ee53931faa1a3992e3f6cb32c5a4ab7" Oct 31 00:47:28.845864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:28.845818 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:28.909985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:28.909873 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:28.988618 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:28.988573 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:47:29.019956 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:29.019873 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:29.051054 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:29.051009 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 31 00:47:29.054456 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:29.053323 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:29.190993 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:29.190931 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" podUID="4ddbafcb-dbbb-48cf-8145-11628800ff7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:33.634937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:33.634869 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" podUID="4ddbafcb-dbbb-48cf-8145-11628800ff7b" containerName="main" containerID="cri-o://e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6" gracePeriod=45 Oct 31 00:47:33.916116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:33.916089 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:47:34.052949 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.052913 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tls-certs\") pod \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " Oct 31 00:47:34.053129 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.052963 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddbafcb-dbbb-48cf-8145-11628800ff7b-kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/4ddbafcb-dbbb-48cf-8145-11628800ff7b-kube-api-access-jbx4z\") pod \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " Oct 31 00:47:34.053129 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.053074 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-model-cache\") pod \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " Oct 31 00:47:34.053256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.053150 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tmp-dir\") pod \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " Oct 31 00:47:34.053256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.053200 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-dshm\") pod \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " Oct 31 00:47:34.053440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.053377 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-home\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-home\") pod \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\" (UID: \"4ddbafcb-dbbb-48cf-8145-11628800ff7b\") " Oct 31 00:47:34.053570 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.053530 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tmp-dir" pod "4ddbafcb-dbbb-48cf-8145-11628800ff7b" (UID: "4ddbafcb-dbbb-48cf-8145-11628800ff7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:34.053819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.053795 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-home" pod "4ddbafcb-dbbb-48cf-8145-11628800ff7b" (UID: "4ddbafcb-dbbb-48cf-8145-11628800ff7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:34.053936 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.053860 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-model-cache" pod "4ddbafcb-dbbb-48cf-8145-11628800ff7b" (UID: "4ddbafcb-dbbb-48cf-8145-11628800ff7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:34.054100 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.054057 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.054100 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.054082 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.054100 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.054097 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.056645 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.056612 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddbafcb-dbbb-48cf-8145-11628800ff7b-kube-api-access-jbx4z" pod "4ddbafcb-dbbb-48cf-8145-11628800ff7b" (UID: "4ddbafcb-dbbb-48cf-8145-11628800ff7b"). InnerVolumeSpecName "kube-api-access-jbx4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:34.058132 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.058104 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-dshm" pod "4ddbafcb-dbbb-48cf-8145-11628800ff7b" (UID: "4ddbafcb-dbbb-48cf-8145-11628800ff7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:34.060920 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.060895 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tls-certs" pod "4ddbafcb-dbbb-48cf-8145-11628800ff7b" (UID: "4ddbafcb-dbbb-48cf-8145-11628800ff7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:34.132122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.132083 5890 generic.go:356] "Generic (PLEG): container finished" podID="4ddbafcb-dbbb-48cf-8145-11628800ff7b" containerID="e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6" exitCode=0 Oct 31 00:47:34.132122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.132122 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" Oct 31 00:47:34.132337 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.132159 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" event={"ID":"4ddbafcb-dbbb-48cf-8145-11628800ff7b","Type":"ContainerDied","Data":"e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6"} Oct 31 00:47:34.132337 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.132202 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns" event={"ID":"4ddbafcb-dbbb-48cf-8145-11628800ff7b","Type":"ContainerDied","Data":"d85b9645018588b32f83d460f61a17e6921f44a197c3a5075412b674ffafde38"} Oct 31 00:47:34.132337 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.132224 5890 scope.go:122] "RemoveContainer" containerID="e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6" Oct 31 00:47:34.143139 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.143121 5890 scope.go:122] "RemoveContainer" containerID="e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6" Oct 31 00:47:34.143445 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:47:34.143414 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6\": container with ID starting with e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6 not found: ID does not exist" containerID="e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6" Oct 31 00:47:34.143571 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.143450 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6"} err="failed to get container status \"e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6\": rpc error: code = NotFound desc = could not find container \"e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6\": container with ID starting with e5449e3287ec46d22d2032c01ca4e37680683a5442c093d7562d55736edc40b6 not found: ID does not exist" Oct 31 00:47:34.155720 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.155695 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddbafcb-dbbb-48cf-8145-11628800ff7b-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.155720 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.155719 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbx4z\" (UniqueName: \"kubernetes.io/projected/4ddbafcb-dbbb-48cf-8145-11628800ff7b-kube-api-access-jbx4z\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.155918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.155731 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddbafcb-dbbb-48cf-8145-11628800ff7b-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.296911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.296874 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns"] Oct 31 00:47:34.299319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:34.299298 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-b-ea66c31d-kserve-5f54ddfc7d-9xtns"] Oct 31 00:47:35.933188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:35.933154 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddbafcb-dbbb-48cf-8145-11628800ff7b" path="/var/lib/kubelet/pods/4ddbafcb-dbbb-48cf-8145-11628800ff7b/volumes" Oct 31 00:47:37.019283 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:47:37.019246 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:47:39.475776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.475735 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" podUID="c0d2c9ee-adaf-4298-a098-466a6489dc4c" containerName="main" containerID="cri-o://9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354" gracePeriod=45 Oct 31 00:47:39.733332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.733275 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:47:39.926692 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.926659 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tls-certs\") pod \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " Oct 31 00:47:39.926889 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.926709 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-home\") pod \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " Oct 31 00:47:39.926889 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.926778 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-dshm\") pod \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " Oct 31 00:47:39.926889 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.926841 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d2c9ee-adaf-4298-a098-466a6489dc4c-kube-api-access-ws9rr\" (UniqueName: \"kubernetes.io/projected/c0d2c9ee-adaf-4298-a098-466a6489dc4c-kube-api-access-ws9rr\") pod \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " Oct 31 00:47:39.927105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.926918 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-model-cache\") pod \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " Oct 31 00:47:39.927105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.926958 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tmp-dir\") pod \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\" (UID: \"c0d2c9ee-adaf-4298-a098-466a6489dc4c\") " Oct 31 00:47:39.927105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.927086 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-home" pod "c0d2c9ee-adaf-4298-a098-466a6489dc4c" (UID: "c0d2c9ee-adaf-4298-a098-466a6489dc4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:39.927271 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.927190 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tmp-dir" pod "c0d2c9ee-adaf-4298-a098-466a6489dc4c" (UID: "c0d2c9ee-adaf-4298-a098-466a6489dc4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:39.927483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.927458 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:39.927483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.927484 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:39.927712 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.927692 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-model-cache" pod "c0d2c9ee-adaf-4298-a098-466a6489dc4c" (UID: "c0d2c9ee-adaf-4298-a098-466a6489dc4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:39.929197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.929169 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d2c9ee-adaf-4298-a098-466a6489dc4c-kube-api-access-ws9rr" pod "c0d2c9ee-adaf-4298-a098-466a6489dc4c" (UID: "c0d2c9ee-adaf-4298-a098-466a6489dc4c"). InnerVolumeSpecName "kube-api-access-ws9rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:39.929303 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.929268 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-dshm" pod "c0d2c9ee-adaf-4298-a098-466a6489dc4c" (UID: "c0d2c9ee-adaf-4298-a098-466a6489dc4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:39.929367 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:39.929341 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tls-certs" pod "c0d2c9ee-adaf-4298-a098-466a6489dc4c" (UID: "c0d2c9ee-adaf-4298-a098-466a6489dc4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:40.028264 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.028189 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d2c9ee-adaf-4298-a098-466a6489dc4c-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:40.028264 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.028216 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:40.028264 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.028226 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws9rr\" (UniqueName: \"kubernetes.io/projected/c0d2c9ee-adaf-4298-a098-466a6489dc4c-kube-api-access-ws9rr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:40.028264 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.028237 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d2c9ee-adaf-4298-a098-466a6489dc4c-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:40.176961 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.176923 5890 generic.go:356] "Generic (PLEG): container finished" podID="c0d2c9ee-adaf-4298-a098-466a6489dc4c" containerID="9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354" exitCode=0 Oct 31 00:47:40.177153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.176970 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" Oct 31 00:47:40.177153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.177010 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" event={"ID":"c0d2c9ee-adaf-4298-a098-466a6489dc4c","Type":"ContainerDied","Data":"9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354"} Oct 31 00:47:40.177153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.177063 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" event={"ID":"c0d2c9ee-adaf-4298-a098-466a6489dc4c","Type":"ContainerDied","Data":"bce2017f473fa1d911446e9ca3d00c3e820c2b73c6b69fe19589e39fff3bc93f"} Oct 31 00:47:40.177153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.177080 5890 scope.go:122] "RemoveContainer" containerID="9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354" Oct 31 00:47:40.186622 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.186602 5890 scope.go:122] "RemoveContainer" containerID="9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354" Oct 31 00:47:40.186917 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:47:40.186896 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354\": container with ID starting with 9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354 not found: ID does not exist" containerID="9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354" Oct 31 00:47:40.187004 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.186927 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354"} err="failed to get container status \"9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354\": rpc error: code = NotFound desc = could not find container \"9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354\": container with ID starting with 9a410fad3aae63340dcc4aae38735871d0230b4ef3a473d6c41ab630e8806354 not found: ID does not exist" Oct 31 00:47:40.338921 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.338841 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6"] Oct 31 00:47:40.386804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.386758 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6"] Oct 31 00:47:40.689428 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:40.689385 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/e2e-conflict-a-ea66c31d-kserve-96b44d688-b5zv6" podUID="c0d2c9ee-adaf-4298-a098-466a6489dc4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 00:47:41.933348 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:41.933312 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d2c9ee-adaf-4298-a098-466a6489dc4c" path="/var/lib/kubelet/pods/c0d2c9ee-adaf-4298-a098-466a6489dc4c/volumes" Oct 31 00:47:59.943366 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.943334 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz"] Oct 31 00:47:59.943927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.943906 5890 cpu_manager.go:414] "containerMap: removing container" podUID="c0d2c9ee-adaf-4298-a098-466a6489dc4c" containerName="main" Oct 31 00:47:59.943976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.943929 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d2c9ee-adaf-4298-a098-466a6489dc4c" containerName="main" Oct 31 00:47:59.943976 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.943971 5890 cpu_manager.go:414] "containerMap: removing container" podUID="4ddbafcb-dbbb-48cf-8145-11628800ff7b" containerName="main" Oct 31 00:47:59.944042 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.943980 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddbafcb-dbbb-48cf-8145-11628800ff7b" containerName="main" Oct 31 00:47:59.944158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.944145 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d2c9ee-adaf-4298-a098-466a6489dc4c" containerName="main" Oct 31 00:47:59.944191 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.944166 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddbafcb-dbbb-48cf-8145-11628800ff7b" containerName="main" Oct 31 00:47:59.948084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:47:59.948068 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.050616 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.050575 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz"] Oct 31 00:48:00.099425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.099391 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rx9\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-kube-api-access-55rx9\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.099611 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.099436 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.099611 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.099458 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31426775-6b80-4ab7-9fb4-bffee896c601-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.099611 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.099554 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.099733 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.099693 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.099872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.099852 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.099917 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.099887 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31426775-6b80-4ab7-9fb4-bffee896c601-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.100048 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.100025 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.100110 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.100072 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.201776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.201703 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.201776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.201737 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.201776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.201773 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rx9\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-kube-api-access-55rx9\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202088 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.201843 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202088 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.201873 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31426775-6b80-4ab7-9fb4-bffee896c601-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202088 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.201899 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202088 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.201958 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202308 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.202117 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202308 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.202141 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31426775-6b80-4ab7-9fb4-bffee896c601-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202308 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.202221 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202308 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.202285 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202488 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.202359 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202488 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.202420 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.202912 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.202890 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31426775-6b80-4ab7-9fb4-bffee896c601-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.204292 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.204270 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.204508 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.204488 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31426775-6b80-4ab7-9fb4-bffee896c601-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.260297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.260261 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.260420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.260321 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rx9\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-kube-api-access-55rx9\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.261590 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.261572 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:00.683868 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.683838 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz"] Oct 31 00:48:00.685493 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:48:00.685462 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31426775_6b80_4ab7_9fb4_bffee896c601.slice/crio-770a703f92ebd2c43d14cf1736c3f26a808f456a9355ca8d76d4a91fe92490c3 WatchSource:0}: Error finding container 770a703f92ebd2c43d14cf1736c3f26a808f456a9355ca8d76d4a91fe92490c3: Status 404 returned error can't find the container with id 770a703f92ebd2c43d14cf1736c3f26a808f456a9355ca8d76d4a91fe92490c3 Oct 31 00:48:00.687760 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.687726 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:00.687834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.687818 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:00.687874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:00.687854 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:01.383890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:01.383856 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" event={"ID":"31426775-6b80-4ab7-9fb4-bffee896c601","Type":"ContainerStarted","Data":"ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22"} Oct 31 00:48:01.384368 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:01.383911 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" event={"ID":"31426775-6b80-4ab7-9fb4-bffee896c601","Type":"ContainerStarted","Data":"770a703f92ebd2c43d14cf1736c3f26a808f456a9355ca8d76d4a91fe92490c3"} Oct 31 00:48:01.500674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:01.500614 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podStartSLOduration=2.500600588 podStartE2EDuration="2.500600588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:01.498743017 +0000 UTC m=+1173.700993452" watchObservedRunningTime="2026-10-31 00:48:01.500600588 +0000 UTC m=+1173.702851020" Oct 31 00:48:02.262022 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:02.261938 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:02.269844 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:02.269663 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:48:02.269844 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:02.269750 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:48:02.399885 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:02.399772 5890 generic.go:356] "Generic (PLEG): container finished" podID="2e885952-4fd9-452a-a737-e5dae9e412a5" containerID="0f8262a9417b2643c1f80d6bd7d8d9cdfc0acd4e463b2272405b1eaab2bc286b" exitCode=0 Oct 31 00:48:02.400359 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:02.399865 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" event={"ID":"2e885952-4fd9-452a-a737-e5dae9e412a5","Type":"ContainerDied","Data":"0f8262a9417b2643c1f80d6bd7d8d9cdfc0acd4e463b2272405b1eaab2bc286b"} Oct 31 00:48:03.265809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.265166 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:48:03.265809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.265227 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:48:03.601165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.601139 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:48:03.642260 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642231 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-trusted-ca-bundle\") pod \"2e885952-4fd9-452a-a737-e5dae9e412a5\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " Oct 31 00:48:03.642420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642285 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-service-ca-bundle\") pod \"2e885952-4fd9-452a-a737-e5dae9e412a5\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " Oct 31 00:48:03.642420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642367 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e885952-4fd9-452a-a737-e5dae9e412a5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2e885952-4fd9-452a-a737-e5dae9e412a5-archives-path\") pod \"2e885952-4fd9-452a-a737-e5dae9e412a5\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " Oct 31 00:48:03.642420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642402 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e885952-4fd9-452a-a737-e5dae9e412a5-kube-api-access-88svn\" (UniqueName: \"kubernetes.io/projected/2e885952-4fd9-452a-a737-e5dae9e412a5-kube-api-access-88svn\") pod \"2e885952-4fd9-452a-a737-e5dae9e412a5\" (UID: \"2e885952-4fd9-452a-a737-e5dae9e412a5\") " Oct 31 00:48:03.642685 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642657 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-service-ca-bundle" pod "2e885952-4fd9-452a-a737-e5dae9e412a5" (UID: "2e885952-4fd9-452a-a737-e5dae9e412a5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:03.642775 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642689 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-trusted-ca-bundle" pod "2e885952-4fd9-452a-a737-e5dae9e412a5" (UID: "2e885952-4fd9-452a-a737-e5dae9e412a5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:03.642775 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642720 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e885952-4fd9-452a-a737-e5dae9e412a5-archives-path" pod "2e885952-4fd9-452a-a737-e5dae9e412a5" (UID: "2e885952-4fd9-452a-a737-e5dae9e412a5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:03.643006 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.642979 5890 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2e885952-4fd9-452a-a737-e5dae9e412a5-archives-path\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:03.643065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.643011 5890 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-trusted-ca-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:03.643065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.643026 5890 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e885952-4fd9-452a-a737-e5dae9e412a5-service-ca-bundle\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:03.671398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.671344 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e885952-4fd9-452a-a737-e5dae9e412a5-kube-api-access-88svn" pod "2e885952-4fd9-452a-a737-e5dae9e412a5" (UID: "2e885952-4fd9-452a-a737-e5dae9e412a5"). InnerVolumeSpecName "kube-api-access-88svn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:03.761031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.743963 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88svn\" (UniqueName: \"kubernetes.io/projected/2e885952-4fd9-452a-a737-e5dae9e412a5-kube-api-access-88svn\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:03.779491 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.779457 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9"] Oct 31 00:48:03.780211 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.780180 5890 cpu_manager.go:414] "containerMap: removing container" podUID="2e885952-4fd9-452a-a737-e5dae9e412a5" containerName="insights-gathering" Oct 31 00:48:03.780211 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.780203 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e885952-4fd9-452a-a737-e5dae9e412a5" containerName="insights-gathering" Oct 31 00:48:03.780392 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.780367 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e885952-4fd9-452a-a737-e5dae9e412a5" containerName="insights-gathering" Oct 31 00:48:03.784091 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.784069 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.834080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.834005 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9"] Oct 31 00:48:03.980568 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980537 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980582 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9h84\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-kube-api-access-l9h84\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980669 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980766 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980704 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980766 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980725 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980766 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980742 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980876 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980824 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980891 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:03.980944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:03.980921 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082333 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082287 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082339 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082371 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082398 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082472 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082575 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082620 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082700 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082728 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.082841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.082745 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9h84\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-kube-api-access-l9h84\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.083257 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.083235 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.083473 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.083458 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.083834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.083746 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.084121 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.084060 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.085513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.085483 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.085862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.085834 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.141589 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.141564 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.141707 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.141663 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9h84\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-kube-api-access-l9h84\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.262815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.262184 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:48:04.262815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.262248 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:48:04.399230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.399152 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:04.416479 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.416395 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" Oct 31 00:48:04.416479 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.416437 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rhclr-8fxxd" event={"ID":"2e885952-4fd9-452a-a737-e5dae9e412a5","Type":"ContainerDied","Data":"1f0fcafdb3b5bec2f0fe79ceb836c955ee56e18a76887333f509ac503dd6048a"} Oct 31 00:48:04.416583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.416492 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f0fcafdb3b5bec2f0fe79ceb836c955ee56e18a76887333f509ac503dd6048a" Oct 31 00:48:04.871515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.871143 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9"] Oct 31 00:48:04.875223 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:48:04.875190 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385cf49c_32b3_4e38_acf7_b2c5ae63dc22.slice/crio-3be1e0db60177986b0602b91d2c97d9140a2fcc850a0c27dec002dd97138dcde WatchSource:0}: Error finding container 3be1e0db60177986b0602b91d2c97d9140a2fcc850a0c27dec002dd97138dcde: Status 404 returned error can't find the container with id 3be1e0db60177986b0602b91d2c97d9140a2fcc850a0c27dec002dd97138dcde Oct 31 00:48:04.878262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.878228 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:04.878599 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.878308 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:04.878599 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:04.878358 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:05.263634 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:05.263200 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:48:05.263634 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:05.263259 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:48:05.426766 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:05.426725 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" event={"ID":"385cf49c-32b3-4e38-acf7-b2c5ae63dc22","Type":"ContainerStarted","Data":"5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b"} Oct 31 00:48:05.426967 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:05.426801 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" event={"ID":"385cf49c-32b3-4e38-acf7-b2c5ae63dc22","Type":"ContainerStarted","Data":"3be1e0db60177986b0602b91d2c97d9140a2fcc850a0c27dec002dd97138dcde"} Oct 31 00:48:05.558867 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:05.558465 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podStartSLOduration=2.558445205 podStartE2EDuration="2.558445205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:05.542364756 +0000 UTC m=+1177.744615189" watchObservedRunningTime="2026-10-31 00:48:05.558445205 +0000 UTC m=+1177.760695641" Oct 31 00:48:06.267808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:06.266903 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:48:06.267808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:06.266968 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:48:06.401165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:06.401127 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:06.404072 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:06.403939 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:48:06.404072 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:06.404012 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:48:07.028039 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:48:07.028012 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:48:07.268954 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:07.268521 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:48:07.268954 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:07.268589 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 00:48:07.400089 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:07.399993 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:48:07.400089 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:07.400061 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:48:08.279256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:08.279101 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:08.279972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:08.279937 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:08.280975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:08.280952 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:48:08.401814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:08.401026 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:48:08.401814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:08.401586 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:48:09.400702 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:09.400612 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:48:09.401096 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:09.400691 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:48:10.400012 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:10.399977 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:48:10.400193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:10.400041 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:48:11.400104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:11.400064 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:48:11.400597 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:11.400118 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 00:48:13.390104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.390061 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf"] Oct 31 00:48:13.395504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.395480 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.440986 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.440952 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:13.441152 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.441024 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:13.447971 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.447932 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf"] Oct 31 00:48:13.457954 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.457919 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"default-dockercfg-fr2qg\"" Oct 31 00:48:13.491060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.491025 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"distinct-w2-29ece5-kserve-self-signed-certs\"" Oct 31 00:48:13.497975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.497948 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"openshift-service-ca.crt\"" Oct 31 00:48:13.498179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.498056 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"kube-root-ca.crt\"" Oct 31 00:48:13.527296 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.527256 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrqs\" (UniqueName: \"kubernetes.io/projected/7b742d5e-5806-4447-b956-c9f443007ea5-kube-api-access-frrqs\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.527485 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.527360 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-home\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.527485 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.527409 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b742d5e-5806-4447-b956-c9f443007ea5-tls-certs\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.527650 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.527616 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-model-cache\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.527761 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.527735 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-dshm\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.527979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.527775 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-tmp-dir\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.618755 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.618722 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:13.619184 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.619155 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:13.620156 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.620137 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:48:13.628627 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.628607 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-model-cache\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.628720 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.628687 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-dshm\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.628776 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.628718 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-tmp-dir\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.628853 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.628831 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrqs\" (UniqueName: \"kubernetes.io/projected/7b742d5e-5806-4447-b956-c9f443007ea5-kube-api-access-frrqs\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.628929 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.628912 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-home\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.628993 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.628944 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b742d5e-5806-4447-b956-c9f443007ea5-tls-certs\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.629103 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.629082 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-model-cache\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.629200 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.629178 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-tmp-dir\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.629286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.629265 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-home\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.631038 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.631007 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-dshm\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.631411 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.631394 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b742d5e-5806-4447-b956-c9f443007ea5-tls-certs\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.687561 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.687532 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrqs\" (UniqueName: \"kubernetes.io/projected/7b742d5e-5806-4447-b956-c9f443007ea5-kube-api-access-frrqs\") pod \"distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:13.717226 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:13.717182 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:14.162688 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:14.162661 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf"] Oct 31 00:48:14.163720 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:48:14.163687 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b742d5e_5806_4447_b956_c9f443007ea5.slice/crio-a37392929bdc65afb2dffa57ee22ac27669a14978e7ad1fc4edc12248a3b3456 WatchSource:0}: Error finding container a37392929bdc65afb2dffa57ee22ac27669a14978e7ad1fc4edc12248a3b3456: Status 404 returned error can't find the container with id a37392929bdc65afb2dffa57ee22ac27669a14978e7ad1fc4edc12248a3b3456 Oct 31 00:48:14.522314 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:14.522280 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" event={"ID":"7b742d5e-5806-4447-b956-c9f443007ea5","Type":"ContainerStarted","Data":"bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85"} Oct 31 00:48:14.522678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:14.522323 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" event={"ID":"7b742d5e-5806-4447-b956-c9f443007ea5","Type":"ContainerStarted","Data":"a37392929bdc65afb2dffa57ee22ac27669a14978e7ad1fc4edc12248a3b3456"} Oct 31 00:48:14.641327 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:14.641258 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podStartSLOduration=1.641244127 podStartE2EDuration="1.641244127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:14.63830128 +0000 UTC m=+1186.840551715" watchObservedRunningTime="2026-10-31 00:48:14.641244127 +0000 UTC m=+1186.843494608" Oct 31 00:48:14.718025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:14.717985 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:23.717707 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:23.717660 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:23.730700 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:23.730669 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:24.618880 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:24.618850 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:27.728809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:27.728050 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.749811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:27.747991 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:48:27.790087 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:27.790041 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:27.790391 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:27.790357 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.854809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:27.841890 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.898809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:27.896186 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:28.002912 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:27.986117 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:28.075808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:28.072611 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:48:28.428810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:28.406435 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:28.512844 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:28.511095 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:48:28.592113 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:28.592074 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:48:28.609858 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:28.608186 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:32.178384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.178346 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx"] Oct 31 00:48:32.184196 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.184164 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.185629 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.185591 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-provider-credentials\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.185778 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.185725 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4l5\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-kube-api-access-9s4l5\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.185938 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.185778 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-routing-overlay-volume\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.185938 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.185882 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-plugins-config-volume\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.237836 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.237796 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx"] Oct 31 00:48:32.281189 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.281153 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 00:48:32.281389 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.281221 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1a4f67076b57\"" Oct 31 00:48:32.281389 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.281298 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 00:48:32.281389 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.281306 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 00:48:32.281389 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.281369 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 00:48:32.282499 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.282480 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zh9bc\"" Oct 31 00:48:32.286736 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.286709 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4l5\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-kube-api-access-9s4l5\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.286855 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.286777 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-routing-overlay-volume\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.286922 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.286856 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-plugins-config-volume\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.286982 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.286946 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-provider-credentials\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.287621 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.287594 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-routing-overlay-volume\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.287741 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.287700 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-plugins-config-volume\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.289542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.289519 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-provider-credentials\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.348582 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.348543 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4l5\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-kube-api-access-9s4l5\") pod \"payload-processing-external-model-5747777fbb-6hbdx\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.499395 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.499313 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:32.877439 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:48:32.877406 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a72cbd_3a1d_4ac9_9f54_9620018d631e.slice/crio-1c1a87780e9a1945afad02bcffc1c92760123f0b442aae43a47ba6bd43d681d9 WatchSource:0}: Error finding container 1c1a87780e9a1945afad02bcffc1c92760123f0b442aae43a47ba6bd43d681d9: Status 404 returned error can't find the container with id 1c1a87780e9a1945afad02bcffc1c92760123f0b442aae43a47ba6bd43d681d9 Oct 31 00:48:32.880116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.880088 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx"] Oct 31 00:48:32.915656 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:32.915624 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" event={"ID":"23a72cbd-3a1d-4ac9-9f54-9620018d631e","Type":"ContainerStarted","Data":"1c1a87780e9a1945afad02bcffc1c92760123f0b442aae43a47ba6bd43d681d9"} Oct 31 00:48:33.924086 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:33.924042 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" event={"ID":"23a72cbd-3a1d-4ac9-9f54-9620018d631e","Type":"ContainerStarted","Data":"6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5"} Oct 31 00:48:33.924507 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:33.924137 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:34.038552 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:34.038492 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" podStartSLOduration=2.038478511 podStartE2EDuration="2.038478511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:34.035927424 +0000 UTC m=+1206.238177858" watchObservedRunningTime="2026-10-31 00:48:34.038478511 +0000 UTC m=+1206.240728955" Oct 31 00:48:36.868613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.868579 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm"] Oct 31 00:48:36.873669 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.873651 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939040 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939003 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d7401-57c1-4cab-9893-8822207e3db6-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939161 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939212 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939361 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939337 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939387 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939413 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq6g\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-kube-api-access-nnq6g\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939494 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939454 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d7401-57c1-4cab-9893-8822207e3db6-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939679 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939560 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.939679 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.939638 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:36.960830 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:36.960796 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm"] Oct 31 00:48:37.022967 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:48:37.022926 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:48:37.040883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.040847 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041074 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.040938 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d7401-57c1-4cab-9893-8822207e3db6-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041074 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041023 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041074 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041065 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041104 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041149 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041172 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq6g\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-kube-api-access-nnq6g\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041203 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d7401-57c1-4cab-9893-8822207e3db6-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041443 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041269 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041443 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041349 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041532 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041754 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d7401-57c1-4cab-9893-8822207e3db6-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041808 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.041944 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.041867 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.043577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.043556 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.043844 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.043826 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d7401-57c1-4cab-9893-8822207e3db6-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.100419 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.100385 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.100577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.100423 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq6g\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-kube-api-access-nnq6g\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.188018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.187984 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:37.610221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.610191 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm"] Oct 31 00:48:37.611412 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:48:37.611383 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682d7401_57c1_4cab_9893_8822207e3db6.slice/crio-4ecf510f1a18ce78c35887749cfacbcac220d76fce7e2630bf0df2567d89e8c6 WatchSource:0}: Error finding container 4ecf510f1a18ce78c35887749cfacbcac220d76fce7e2630bf0df2567d89e8c6: Status 404 returned error can't find the container with id 4ecf510f1a18ce78c35887749cfacbcac220d76fce7e2630bf0df2567d89e8c6 Oct 31 00:48:37.613973 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.613941 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:37.614060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.614005 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:37.614060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.614034 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:37.957873 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.957825 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" event={"ID":"682d7401-57c1-4cab-9893-8822207e3db6","Type":"ContainerStarted","Data":"92c28d01a9259b07002df5ea6ef5a7f492a5cd1e405511d49b706380bf56c2c9"} Oct 31 00:48:37.958264 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:37.957899 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" event={"ID":"682d7401-57c1-4cab-9893-8822207e3db6","Type":"ContainerStarted","Data":"4ecf510f1a18ce78c35887749cfacbcac220d76fce7e2630bf0df2567d89e8c6"} Oct 31 00:48:38.087303 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:38.087242 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podStartSLOduration=2.08722858 podStartE2EDuration="2.08722858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:38.084107436 +0000 UTC m=+1210.286357880" watchObservedRunningTime="2026-10-31 00:48:38.08722858 +0000 UTC m=+1210.289479012" Oct 31 00:48:38.192025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:38.191982 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:38.194030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:38.193995 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:48:38.194188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:38.194058 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 00:48:39.188962 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:39.188922 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:48:39.189372 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:39.189001 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 00:48:40.188906 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:40.188866 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:48:40.189081 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:40.188927 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 00:48:41.188419 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:41.188377 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:48:41.188611 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:41.188442 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 00:48:42.188872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:42.188828 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:48:42.189386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:42.188897 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 00:48:43.188688 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:43.188650 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:48:43.188937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:43.188712 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 00:48:44.189343 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:44.189306 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:48:44.189715 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:44.189380 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 00:48:45.017410 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:44.987993 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd"] Oct 31 00:48:45.019638 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.019605 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" podUID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:45.024869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.024772 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.075975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.075934 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd"] Oct 31 00:48:45.163007 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.162965 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163257 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163029 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163257 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163172 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163257 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163226 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163464 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163386 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163464 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163434 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163581 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163484 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whksz\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-kube-api-access-whksz\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163640 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163596 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.163698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.163651 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.200535 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.200497 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:45.201627 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.201597 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:45.202748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.202724 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:48:45.264725 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.264683 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.264935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.264769 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.264935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.264836 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whksz\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-kube-api-access-whksz\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.264935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.264906 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.264968 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265035 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265115 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265271 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265150 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265271 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265174 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265271 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265179 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265423 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265268 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265683 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265648 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.265900 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.265860 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.266138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.266118 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.268496 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.268433 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.269226 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.269184 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.324482 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.324447 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.324688 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.324666 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whksz\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-kube-api-access-whksz\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.343422 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.343395 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:45.768798 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:48:45.768752 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c2e460_a5c1_4aa7_adce_52feeb05781c.slice/crio-60ee9d69649a59e9a3e7dbc1854541936fa3b3b7f04579f37ef17556c71a9cf8 WatchSource:0}: Error finding container 60ee9d69649a59e9a3e7dbc1854541936fa3b3b7f04579f37ef17556c71a9cf8: Status 404 returned error can't find the container with id 60ee9d69649a59e9a3e7dbc1854541936fa3b3b7f04579f37ef17556c71a9cf8 Oct 31 00:48:45.769121 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.769095 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd"] Oct 31 00:48:45.771618 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.771577 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:45.771714 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.771664 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:45.771756 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:45.771710 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:46.042930 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.042832 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" event={"ID":"e5c2e460-a5c1-4aa7-adce-52feeb05781c","Type":"ContainerStarted","Data":"a495e02547d6e9f9367022a59906813082eb8d6bfa899413936b22cbbf486894"} Oct 31 00:48:46.042930 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.042908 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" event={"ID":"e5c2e460-a5c1-4aa7-adce-52feeb05781c","Type":"ContainerStarted","Data":"60ee9d69649a59e9a3e7dbc1854541936fa3b3b7f04579f37ef17556c71a9cf8"} Oct 31 00:48:46.201652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.201570 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podStartSLOduration=2.201524817 podStartE2EDuration="2.201524817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:46.188946641 +0000 UTC m=+1218.391197098" watchObservedRunningTime="2026-10-31 00:48:46.201524817 +0000 UTC m=+1218.403775252" Oct 31 00:48:46.344445 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.344351 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:46.346330 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.346299 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:46.346446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.346357 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:46.448247 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.448008 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx"] Oct 31 00:48:46.450514 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:46.450478 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" podUID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:47.349221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:47.349183 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:47.349840 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:47.349809 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:48.353809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:48.348165 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:48.353809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:48.348228 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:49.139066 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.139031 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j"] Oct 31 00:48:49.142843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.142826 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.233314 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.233274 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.233491 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.233329 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.233491 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.233464 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27bn\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-kube-api-access-t27bn\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.233634 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.233613 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0931745-893d-45b3-950d-1758093cc8ac-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.233696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.233669 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.233774 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.233747 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0931745-893d-45b3-950d-1758093cc8ac-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.233873 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.233856 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.234034 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.234010 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.234135 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.234118 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335502 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335540 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335548 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335566 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335584 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27bn\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-kube-api-access-t27bn\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335661 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0931745-893d-45b3-950d-1758093cc8ac-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335700 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335730 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0931745-893d-45b3-950d-1758093cc8ac-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.335802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335779 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.336121 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.335884 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.336121 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.336080 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.336121 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.336112 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.336334 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.336308 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.336407 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.336320 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.336719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.336694 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0931745-893d-45b3-950d-1758093cc8ac-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.338290 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.338261 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.338450 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.338429 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0931745-893d-45b3-950d-1758093cc8ac-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.391288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.391194 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:49.391288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.391262 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:49.405073 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.405037 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27bn\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-kube-api-access-t27bn\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.405947 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.405918 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j"] Oct 31 00:48:49.407468 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.407443 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.482182 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.482139 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:49.919805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.918047 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j"] Oct 31 00:48:49.957809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.951638 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:49.957809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.951758 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:49.957809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:49.951830 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:50.227995 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:50.227962 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" event={"ID":"b0931745-893d-45b3-950d-1758093cc8ac","Type":"ContainerStarted","Data":"6285c272b318d38357df2cbfba6db0f48021c092ed7bf1678a9b86f48afcef0d"} Oct 31 00:48:50.359009 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:50.345550 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:50.359009 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:50.345606 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:51.298921 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.298845 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" event={"ID":"b0931745-893d-45b3-950d-1758093cc8ac","Type":"ContainerStarted","Data":"4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4"} Oct 31 00:48:51.346811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.346379 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:51.346811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.346456 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:51.452330 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.450913 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" podUID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" containerName="payload-processing" containerID="cri-o://6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5" gracePeriod=25 Oct 31 00:48:51.499126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.495896 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:48:51.499126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.496449 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:51.499126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.496513 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:51.499126 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:48:51.496841 5890 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a72cbd_3a1d_4ac9_9f54_9620018d631e.slice/crio-6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5.scope/cpu.weight": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a72cbd_3a1d_4ac9_9f54_9620018d631e.slice/crio-6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5.scope/cpu.weight: no such device Oct 31 00:48:51.794141 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.790871 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:51.865018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.843091 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 00:48:51.921266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.921223 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:51.994393 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:51.994345 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.037971 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.037909 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:48:52.066488 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.066439 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.179521 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:48:52.179475 5890 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a72cbd_3a1d_4ac9_9f54_9620018d631e.slice/crio-6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a72cbd_3a1d_4ac9_9f54_9620018d631e.slice/crio-conmon-6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:48:52.497872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.444728 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.497872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.446006 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:52.497872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.446048 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:52.497872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.491428 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:52.497872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.491487 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:52.573811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.573734 5890 generic.go:356] "Generic (PLEG): container finished" podID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" containerID="6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5" exitCode=0 Oct 31 00:48:52.575320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.575285 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" event={"ID":"23a72cbd-3a1d-4ac9-9f54-9620018d631e","Type":"ContainerDied","Data":"6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5"} Oct 31 00:48:52.640030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.617968 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.832900 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.806843 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.832900 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.813207 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:48:52.971343 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.971286 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 00:48:52.991853 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.974553 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 00:48:52.993542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.993501 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:48:52.993771 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:52.993745 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:48:53.021203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.001510 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:53.021203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.013094 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:48:53.095187 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.040852 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:53.095187 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.066349 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:48:53.095187 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.068099 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:48:53.095187 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.088599 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podStartSLOduration=4.088575865 podStartE2EDuration="4.088575865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:51.549887308 +0000 UTC m=+1223.752137759" watchObservedRunningTime="2026-10-31 00:48:53.088575865 +0000 UTC m=+1225.290826300" Oct 31 00:48:53.097612 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.096708 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:48:53.409862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.409706 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:53.409862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.409763 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:53.591041 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.585425 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:53.591041 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.585481 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:53.751810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.747327 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:53.751810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.747736 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:53.832812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.830631 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:48:53.832812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.831141 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:48:53.997060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:53.997011 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.008866 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.004933 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:48:54.031811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.001232 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.100758 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.100715 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.172850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.132308 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:54.461807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.429252 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-routing-overlay-volume\") pod \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " Oct 31 00:48:54.472005 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.471967 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-kube-api-access-9s4l5\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-kube-api-access-9s4l5\") pod \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " Oct 31 00:48:54.506809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.503480 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-plugins-config-volume\") pod \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " Oct 31 00:48:54.567542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.440840 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-routing-overlay-volume" pod "23a72cbd-3a1d-4ac9-9f54-9620018d631e" (UID: "23a72cbd-3a1d-4ac9-9f54-9620018d631e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:54.567779 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.456577 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:54.571860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.567939 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:54.572108 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.511450 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-plugins-config-volume" pod "23a72cbd-3a1d-4ac9-9f54-9620018d631e" (UID: "23a72cbd-3a1d-4ac9-9f54-9620018d631e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:54.658811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.633864 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-provider-credentials\") pod \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\" (UID: \"23a72cbd-3a1d-4ac9-9f54-9620018d631e\") " Oct 31 00:48:54.658811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.635187 5890 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-routing-overlay-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:54.658811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.635241 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a72cbd-3a1d-4ac9-9f54-9620018d631e-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:54.777969 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.777835 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.048860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.779485 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 31 00:48:55.048860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.836421 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:48:55.048860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:54.836471 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:48:55.049258 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:48:54.951143 5890 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.011s" Oct 31 00:48:55.049258 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.049241 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" event={"ID":"23a72cbd-3a1d-4ac9-9f54-9620018d631e","Type":"ContainerDied","Data":"1c1a87780e9a1945afad02bcffc1c92760123f0b442aae43a47ba6bd43d681d9"} Oct 31 00:48:55.049366 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.049318 5890 scope.go:122] "RemoveContainer" containerID="6e18d73ae786e6f81197e2955c8d21158d4bdbaf68d5291f5619b7aee6c0f4f5" Oct 31 00:48:55.061834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.011127 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.061834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.047722 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:48:55.085295 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.064308 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.085295 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.074909 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:55.085295 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.074978 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:55.128805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.127918 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.157635 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.157595 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-kube-api-access-9s4l5" pod "23a72cbd-3a1d-4ac9-9f54-9620018d631e" (UID: "23a72cbd-3a1d-4ac9-9f54-9620018d631e"). InnerVolumeSpecName "kube-api-access-9s4l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:55.256074 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.256020 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-provider-credentials" pod "23a72cbd-3a1d-4ac9-9f54-9620018d631e" (UID: "23a72cbd-3a1d-4ac9-9f54-9620018d631e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:55.259405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.259367 5890 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-provider-credentials\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:55.259625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.259607 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4l5\" (UniqueName: \"kubernetes.io/projected/23a72cbd-3a1d-4ac9-9f54-9620018d631e-kube-api-access-9s4l5\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:55.367997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.367957 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:48:55.368282 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.368248 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 00:48:55.611882 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.609617 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:55.611882 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.609714 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:55.805809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.792310 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx" Oct 31 00:48:55.961809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:55.960146 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 00:48:56.049923 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.049878 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:48:56.050538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.050502 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 31 00:48:56.051007 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.050974 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:48:56.076808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.051384 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:48:56.102814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.067177 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 00:48:56.118813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.103087 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:56.130393 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.130351 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:48:56.427799 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.423970 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:48:56.438848 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.437879 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx"] Oct 31 00:48:56.516320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.516227 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:56.525809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.516920 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:56.526069 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.522071 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:48:56.640230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.640197 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5747777fbb-6hbdx"] Oct 31 00:48:56.875156 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.875049 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:48:56.941388 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.941352 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:48:56.985808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.985039 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:48:56.988568 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.988540 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:48:56.988812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.988797 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:48:57.005822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:56.994907 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:48:57.184722 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:57.184692 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:57.185657 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:57.185631 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:57.187548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:57.187428 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 00:48:57.483387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:57.483305 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:57.483387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:57.483370 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:57.936471 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:57.936438 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" path="/var/lib/kubelet/pods/23a72cbd-3a1d-4ac9-9f54-9620018d631e/volumes" Oct 31 00:48:58.482517 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:58.482483 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:58.482838 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:58.482768 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:59.482925 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:59.482878 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:48:59.483472 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:59.482948 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:48:59.945966 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:59.945930 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5"] Oct 31 00:48:59.946567 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:59.946547 5890 cpu_manager.go:414] "containerMap: removing container" podUID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" containerName="payload-processing" Oct 31 00:48:59.946664 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:59.946569 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" containerName="payload-processing" Oct 31 00:48:59.946768 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:59.946753 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a72cbd-3a1d-4ac9-9f54-9620018d631e" containerName="payload-processing" Oct 31 00:48:59.951780 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:48:59.951760 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.009311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.009276 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5"] Oct 31 00:49:00.039516 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.039480 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.039718 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.039576 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.039718 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.039612 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.039881 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.039753 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.039881 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.039819 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.040354 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.040330 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.040534 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.040516 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglvm\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-kube-api-access-nglvm\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.040663 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.040648 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.040826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.040776 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142049 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglvm\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-kube-api-access-nglvm\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142309 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142132 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142309 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142161 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142309 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142259 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142471 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142325 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142471 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142358 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142576 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142484 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142576 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142518 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142679 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142576 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142679 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142628 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142701 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142766 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.142925 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.142852 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.143286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.143263 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.145089 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.145062 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.145216 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.145188 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.201326 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.201244 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.201475 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.201437 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglvm\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-kube-api-access-nglvm\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.265996 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.265957 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:00.482988 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.482901 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:49:00.482988 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.482959 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 00:49:00.693726 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:49:00.693696 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf555bceb_b443_4267_b3e9_deaae7ad2e1a.slice/crio-8d216c80be7118387f707c3e42b0f7e2ff8c8a48eb9d5d4dbf1938bfad77b703 WatchSource:0}: Error finding container 8d216c80be7118387f707c3e42b0f7e2ff8c8a48eb9d5d4dbf1938bfad77b703: Status 404 returned error can't find the container with id 8d216c80be7118387f707c3e42b0f7e2ff8c8a48eb9d5d4dbf1938bfad77b703 Oct 31 00:49:00.694136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.694109 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5"] Oct 31 00:49:00.696493 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.696454 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:00.696571 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.696544 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:00.696607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:00.696589 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:01.017277 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.015122 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" event={"ID":"f555bceb-b443-4267-b3e9-deaae7ad2e1a","Type":"ContainerStarted","Data":"1f2a2b042d65fa1c2d9c1200e9367641b4f5db2b2fae4cd6ac94eec16fa42d56"} Oct 31 00:49:01.017277 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.015194 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" event={"ID":"f555bceb-b443-4267-b3e9-deaae7ad2e1a","Type":"ContainerStarted","Data":"8d216c80be7118387f707c3e42b0f7e2ff8c8a48eb9d5d4dbf1938bfad77b703"} Oct 31 00:49:01.135823 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.135719 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podStartSLOduration=2.135701432 podStartE2EDuration="2.135701432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:01.133918659 +0000 UTC m=+1233.336169095" watchObservedRunningTime="2026-10-31 00:49:01.135701432 +0000 UTC m=+1233.337951857" Oct 31 00:49:01.267862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.267142 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:01.277035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.276991 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:49:01.277373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.277326 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:49:01.488558 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.487502 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:49:01.488558 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.488022 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:49:01.489073 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:01.489054 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:49:02.267207 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.267164 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:49:02.267558 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.267521 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:49:02.478900 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.478845 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw"] Oct 31 00:49:02.485562 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.485529 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.494777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.494741 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-home\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.495205 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.494969 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tmp-dir\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.495205 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.495053 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-model-cache\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.495205 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.495089 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-dshm\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.495205 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.495130 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tls-certs\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.495427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.495217 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzc7w\" (UniqueName: \"kubernetes.io/projected/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-kube-api-access-fzc7w\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.514224 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.514189 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw"] Oct 31 00:49:02.536127 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.536034 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"distinct-2-w2-29ece5-kserve-self-signed-certs\"" Oct 31 00:49:02.597048 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.597002 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzc7w\" (UniqueName: \"kubernetes.io/projected/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-kube-api-access-fzc7w\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.597048 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.597070 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-home\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.603515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.598199 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-home\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.603515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.598619 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tmp-dir\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.603515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.598736 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-model-cache\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.603515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.598775 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-dshm\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.603515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.598844 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tls-certs\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.604233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.604053 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tmp-dir\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.607888 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.604537 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-model-cache\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.607888 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.607320 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-dshm\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.610530 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.610490 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tls-certs\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.688863 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.687644 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzc7w\" (UniqueName: \"kubernetes.io/projected/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-kube-api-access-fzc7w\") pod \"distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:02.801746 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:02.801655 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:03.269286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:03.269249 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:49:03.269483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:03.269309 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:49:03.316572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:03.316541 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw"] Oct 31 00:49:03.317666 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:49:03.317641 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f49a78e_3845_4cf1_97d0_1d6ea81b3d60.slice/crio-5e6caefe9ccc10ef3e17a52edcd87f9c49e3adc937af6ec70c52c8029d3b7274 WatchSource:0}: Error finding container 5e6caefe9ccc10ef3e17a52edcd87f9c49e3adc937af6ec70c52c8029d3b7274: Status 404 returned error can't find the container with id 5e6caefe9ccc10ef3e17a52edcd87f9c49e3adc937af6ec70c52c8029d3b7274 Oct 31 00:49:04.082513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:04.080143 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" event={"ID":"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60","Type":"ContainerStarted","Data":"fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc"} Oct 31 00:49:04.082513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:04.080203 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" event={"ID":"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60","Type":"ContainerStarted","Data":"5e6caefe9ccc10ef3e17a52edcd87f9c49e3adc937af6ec70c52c8029d3b7274"} Oct 31 00:49:04.194754 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:04.194679 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podStartSLOduration=2.19466506 podStartE2EDuration="2.19466506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:04.19286557 +0000 UTC m=+1236.395116004" watchObservedRunningTime="2026-10-31 00:49:04.19466506 +0000 UTC m=+1236.396915532" Oct 31 00:49:04.266810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:04.266426 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:49:04.266810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:04.266495 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:49:04.802221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:04.802184 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:05.269266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:05.269223 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:49:05.269692 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:05.269294 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:49:06.268808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:06.268148 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:49:06.268808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:06.268209 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:49:07.024895 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:49:07.024853 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:49:07.266757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:07.266720 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 00:49:07.266982 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:07.266773 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 00:49:08.270695 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:08.270664 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:08.271165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:08.271084 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:08.271867 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:08.271844 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:49:12.807809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:12.806771 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:13.825819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:13.824903 5890 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:13.843541 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:13.836911 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.025813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.023881 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.025813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.024670 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:49:14.059935 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.059880 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.063932 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.061307 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 00:49:14.063932 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.061516 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.169867 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.157357 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.692810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.691007 5890 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:14.692810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.691113 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.731953 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.731884 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.889828 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:14.840296 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.003452 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.003360 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:49:15.026153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.026102 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.026745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.026394 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.027565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.026978 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:49:15.027761 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.027746 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:49:15.062957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.062894 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.112810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.063219 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.140937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.063256 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 00:49:15.142967 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.114542 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:49:15.143262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.143244 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:49:15.143448 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.143426 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:49:15.167818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.162927 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.169243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.168391 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:49:15.176817 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.174950 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:15.176817 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.175013 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.916806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:15.903014 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.009243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.007221 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.034919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.034868 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:49:16.071823 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.038977 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:49:16.145321 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.109332 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 31 00:49:16.145321 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.134582 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 00:49:16.188874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.178074 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.188874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.179864 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:49:16.427992 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.427939 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:16.428164 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:16.428027 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.017950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.017899 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.046030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.045905 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:17.070842 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.046287 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.071151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.041104 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.096419 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.096366 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.136845 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.115892 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.139193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.139147 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.202933 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.202879 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 00:49:17.204872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.203195 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.746933 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:17.746883 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.085212 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.085092 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.107010 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.106944 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.132025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.131914 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.180799 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.180575 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:49:18.181518 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.181479 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 00:49:18.209945 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.209885 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.223808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.216110 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.416660 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.416578 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:49:18.428802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.427848 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.433821 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.433733 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:49:18.436079 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.436060 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:49:18.696172 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.696128 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:49:18.741975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.741928 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:18.742275 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.742243 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:18.758806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:18.746232 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:49:19.115059 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:19.114919 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 31 00:49:19.210680 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:19.210636 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:19.219143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:19.219098 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:19.263089 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:19.263058 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:49:19.539131 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:19.538164 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:49:22.814293 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:22.814262 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:22.825526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:22.825497 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:29.944901 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:29.944863 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b"] Oct 31 00:49:29.950281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:29.950256 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.007469 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.007434 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b"] Oct 31 00:49:30.091467 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091420 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.091675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091485 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.091675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091542 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rtk\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-kube-api-access-r7rtk\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.091675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091612 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.091878 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091717 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.091878 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091750 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.091878 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091808 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.092041 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.091975 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.092113 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.092054 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.193620 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193588 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.193620 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193621 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.193904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193644 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.193904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193744 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.193904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193837 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.193904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193903 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193928 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.193976 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rtk\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-kube-api-access-r7rtk\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.194025 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194122 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.194103 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194347 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.194132 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194347 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.194170 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194447 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.194345 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.194775 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.194753 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.196262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.196210 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.196448 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.196429 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.254937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.254905 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.255192 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.255168 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rtk\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-kube-api-access-r7rtk\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.264017 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.263990 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:30.696495 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.696464 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b"] Oct 31 00:49:30.697447 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:49:30.697412 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1da0533_8d2e_4d41_aa09_10c4f66e84a4.slice/crio-697e88e609d38d3c45251fe866f75777e45953e4e5d759b839a7a0ed90b4b2f0 WatchSource:0}: Error finding container 697e88e609d38d3c45251fe866f75777e45953e4e5d759b839a7a0ed90b4b2f0: Status 404 returned error can't find the container with id 697e88e609d38d3c45251fe866f75777e45953e4e5d759b839a7a0ed90b4b2f0 Oct 31 00:49:30.699809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.699749 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:30.699919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.699846 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:30.699919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:30.699892 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:31.137675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:31.137560 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" event={"ID":"a1da0533-8d2e-4d41-aa09-10c4f66e84a4","Type":"ContainerStarted","Data":"807a0c7bd0facd8384cf69283db924ad460390ac6a0364d2d25cbdc7863f63c1"} Oct 31 00:49:31.137675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:31.137647 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" event={"ID":"a1da0533-8d2e-4d41-aa09-10c4f66e84a4","Type":"ContainerStarted","Data":"697e88e609d38d3c45251fe866f75777e45953e4e5d759b839a7a0ed90b4b2f0"} Oct 31 00:49:31.266811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:31.261514 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podStartSLOduration=2.2614940150000002 podStartE2EDuration="2.261494015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:31.260256416 +0000 UTC m=+1263.462506862" watchObservedRunningTime="2026-10-31 00:49:31.261494015 +0000 UTC m=+1263.463744449" Oct 31 00:49:31.274279 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:31.274192 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:31.278565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:31.278387 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:49:31.278565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:31.278442 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 00:49:32.270806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:32.268933 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:49:32.270806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:32.269013 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 00:49:33.270810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:33.269669 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:49:33.270810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:33.269741 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 00:49:34.268808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:34.266157 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:49:34.268808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:34.266233 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 00:49:35.266206 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:35.266167 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:49:35.266826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:35.266766 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 00:49:36.271809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:36.267890 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:49:36.271809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:36.267969 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 00:49:37.022810 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:49:37.022404 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:49:37.270687 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:37.270650 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:49:37.271028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:37.270989 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 00:49:39.312971 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:39.312929 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:39.313575 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:39.313540 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.648869 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:39.639400 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.817387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:39.817341 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.052052 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.050923 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:49:40.052052 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.051738 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.091748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.091698 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:49:40.115659 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.114860 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.154153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.154102 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 00:49:40.183703 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.182889 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.383866 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.359165 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:40.408923 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.408873 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.419929 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.419885 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.861286 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.861240 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 00:49:40.945297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:40.945260 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:41.074810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.067232 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:49:41.103341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.103310 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:49:41.104435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.077327 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:49:41.105564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.101947 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.106277 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.105728 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:49:41.106379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.106301 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:49:41.123917 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.116974 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.129819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.124451 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:49:41.198309 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.198254 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.199188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.199163 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:49:41.468808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.440882 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.468808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.441487 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:49:41.496894 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.496857 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:41.497163 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.497131 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.549014 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.548959 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:41.549468 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:41.549318 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.065810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.044540 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 00:49:42.110808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.107128 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.110808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.107595 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:49:42.110808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.108823 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:49:42.176809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.175592 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.176809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.176167 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 31 00:49:42.239847 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.214203 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.343230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.343130 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:49:42.398904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.398867 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb"] Oct 31 00:49:42.445870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.440903 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:42.458947 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.458902 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 00:49:42.460560 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.460523 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 31 00:49:42.466817 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.465472 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb"] Oct 31 00:49:42.521207 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.521171 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7e6fe88b-dockercfg-8mbks\"" Oct 31 00:49:42.545771 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.545726 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftkh\" (UniqueName: \"kubernetes.io/projected/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-kube-api-access-mftkh\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:42.567816 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.564512 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:42.621871 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.618311 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7e6fe88b\"" Oct 31 00:49:42.661595 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.656715 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:42.661595 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.656778 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.686015 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.685976 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftkh\" (UniqueName: \"kubernetes.io/projected/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-kube-api-access-mftkh\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:42.716030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.715990 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:42.748559 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:42.748147 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:43.023151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.023121 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:49:43.064291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.048951 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 31 00:49:43.118708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.118663 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:43.119301 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.119272 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:49:43.139686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.135452 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftkh\" (UniqueName: \"kubernetes.io/projected/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-kube-api-access-mftkh\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:43.196804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.188890 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:43.196804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.189286 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 31 00:49:43.196804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.189594 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:49:43.293874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.284006 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:43.461919 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.461869 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:43.504344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.504312 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:49:43.518835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.518808 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:49:43.552769 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.552740 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:49:43.564767 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.564738 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:49:43.610828 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.586076 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:43.644354 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.644314 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:43.653978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.653910 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 00:49:43.815870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.797582 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:49:43.815870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:43.800483 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:49:44.096172 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:49:44.096131 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99c6dd3_2417_44c0_b12c_9a5ea790b6e7.slice/crio-56e2e764358b3e3257e743ded853393398d6ff43cd64ea4783d322fc57536a08 WatchSource:0}: Error finding container 56e2e764358b3e3257e743ded853393398d6ff43cd64ea4783d322fc57536a08: Status 404 returned error can't find the container with id 56e2e764358b3e3257e743ded853393398d6ff43cd64ea4783d322fc57536a08 Oct 31 00:49:44.098629 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.098602 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb"] Oct 31 00:49:44.144635 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.144588 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz"] Oct 31 00:49:44.153390 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.153355 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.184270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.184210 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04083cb-7d47-4d40-8b31-76005517baa1-tls-certs\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.184418 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.184321 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-home\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.184418 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.184392 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-tmp-dir\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.184554 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.184526 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-dshm\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.184673 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.184568 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-model-cache\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.184673 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.184637 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jr7t\" (UniqueName: \"kubernetes.io/projected/d04083cb-7d47-4d40-8b31-76005517baa1-kube-api-access-4jr7t\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.195189 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.195157 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz"] Oct 31 00:49:44.203964 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.203935 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"kube-root-ca.crt\"" Oct 31 00:49:44.204215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.204193 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"openshift-service-ca.crt\"" Oct 31 00:49:44.204379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.204337 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"default-dockercfg-mkgsq\"" Oct 31 00:49:44.204499 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.204476 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"distinct-w1-b63c24-kserve-self-signed-certs\"" Oct 31 00:49:44.285639 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.285593 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-home\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.285849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.285675 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-tmp-dir\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.285849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.285734 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-dshm\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.285849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.285753 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-model-cache\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.285849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.285814 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jr7t\" (UniqueName: \"kubernetes.io/projected/d04083cb-7d47-4d40-8b31-76005517baa1-kube-api-access-4jr7t\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.286080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.285896 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04083cb-7d47-4d40-8b31-76005517baa1-tls-certs\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.286080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.286064 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-tmp-dir\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.286182 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.286064 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-home\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.286230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.286202 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-model-cache\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.288072 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.288046 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-dshm\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.288301 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.288283 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04083cb-7d47-4d40-8b31-76005517baa1-tls-certs\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.348863 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.348736 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jr7t\" (UniqueName: \"kubernetes.io/projected/d04083cb-7d47-4d40-8b31-76005517baa1-kube-api-access-4jr7t\") pod \"distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.472985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.472941 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:44.864446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.864414 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz"] Oct 31 00:49:44.865645 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:49:44.865613 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd04083cb_7d47_4d40_8b31_76005517baa1.slice/crio-34a9248783da084bd354df807656b1113a554f668d6c39c5e1013d79835d9ca8 WatchSource:0}: Error finding container 34a9248783da084bd354df807656b1113a554f668d6c39c5e1013d79835d9ca8: Status 404 returned error can't find the container with id 34a9248783da084bd354df807656b1113a554f668d6c39c5e1013d79835d9ca8 Oct 31 00:49:44.900939 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.900884 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" event={"ID":"d04083cb-7d47-4d40-8b31-76005517baa1","Type":"ContainerStarted","Data":"34a9248783da084bd354df807656b1113a554f668d6c39c5e1013d79835d9ca8"} Oct 31 00:49:44.905117 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.905058 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" event={"ID":"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7","Type":"ContainerStarted","Data":"4940e62556d41c7db9a5c20d67235e2415a80346d8574936f521b00f5534dc56"} Oct 31 00:49:44.905272 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.905150 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" event={"ID":"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7","Type":"ContainerStarted","Data":"56e2e764358b3e3257e743ded853393398d6ff43cd64ea4783d322fc57536a08"} Oct 31 00:49:44.905272 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:44.905205 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:49:45.017298 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:45.017247 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" podStartSLOduration=4.017231169 podStartE2EDuration="4.017231169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:45.017213352 +0000 UTC m=+1277.219463790" watchObservedRunningTime="2026-10-31 00:49:45.017231169 +0000 UTC m=+1277.219481602" Oct 31 00:49:45.913321 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:45.913276 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" event={"ID":"d04083cb-7d47-4d40-8b31-76005517baa1","Type":"ContainerStarted","Data":"a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c"} Oct 31 00:49:46.028388 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:46.028335 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" podStartSLOduration=2.028321364 podStartE2EDuration="2.028321364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:46.025358524 +0000 UTC m=+1278.227608958" watchObservedRunningTime="2026-10-31 00:49:46.028321364 +0000 UTC m=+1278.230571857" Oct 31 00:49:46.474232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:46.474194 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:54.474030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:54.473989 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:54.486522 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:54.486489 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:54.984762 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:54.984727 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:49:55.916418 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:49:55.916392 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:50:07.019437 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:50:07.019398 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:50:12.438982 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:12.438928 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:12.535810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:12.533244 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:12.571808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:12.569016 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:12.805871 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:12.805017 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.047808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.046958 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.190864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.116090 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.190864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.116536 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 00:50:13.190864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.117262 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.320911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.320863 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p"] Oct 31 00:50:13.789739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.789690 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.809979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.806307 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 31 00:50:13.817809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.815984 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.830811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.823855 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.879810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:13.874882 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:50:14.013809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.004893 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.037812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.024333 5890 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:50:14.037812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.024416 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.063774 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.050263 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.113839 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.091660 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:50:14.113839 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.091739 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.113839 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.104897 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p"] Oct 31 00:50:14.116310 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.116273 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.117872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.117843 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.132523 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.130369 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.219425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.211140 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7e6fe88b-serving-cert\"" Oct 31 00:50:14.367890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.343824 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:50:14.367890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.343895 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:50:14.367890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.343945 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:50:14.367890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.343979 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:50:14.367890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.344013 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:50:14.367890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.344045 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:50:14.573571 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.434699 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthd9\" (UniqueName: \"kubernetes.io/projected/913d0944-09fa-4e45-8373-ebbe5f0f1bca-kube-api-access-rthd9\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.575048 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.575016 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-maas-api-tls\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.575422 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.575400 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-metrics-certs\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.575696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.575674 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913d0944-09fa-4e45-8373-ebbe5f0f1bca-openshift-service-ca\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.731810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.706617 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-maas-api-tls\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.737808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.732176 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-metrics-certs\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.742404 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.738115 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913d0944-09fa-4e45-8373-ebbe5f0f1bca-openshift-service-ca\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.804179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.804136 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthd9\" (UniqueName: \"kubernetes.io/projected/913d0944-09fa-4e45-8373-ebbe5f0f1bca-kube-api-access-rthd9\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.804900 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.769977 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913d0944-09fa-4e45-8373-ebbe5f0f1bca-openshift-service-ca\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.805059 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.768277 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.843953 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.843918 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-metrics-certs\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.951691 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.844653 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-maas-api-tls\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:14.951999 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.925998 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 00:50:14.952429 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.926962 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 00:50:14.952810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.950577 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:14.973814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.953218 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:50:14.975987 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:14.975541 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" podUID="b661f84e-8c0e-484f-a531-afc81020e2f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:15.061895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.041701 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthd9\" (UniqueName: \"kubernetes.io/projected/913d0944-09fa-4e45-8373-ebbe5f0f1bca-kube-api-access-rthd9\") pod \"maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:15.072931 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.072883 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:15.083813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.073437 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 00:50:15.088811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.084391 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:50:15.144819 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.134294 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 00:50:15.183809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.161774 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 00:50:15.183809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.162119 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 31 00:50:15.230348 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.230315 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 00:50:15.246113 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.246077 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:15.376857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.351889 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-b687k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:50:15.376857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.351943 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:15.485805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.468298 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 00:50:15.485805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.469445 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 00:50:15.499894 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.499329 5890 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:15.542209 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.542178 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 00:50:15.684838 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:15.670205 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2" Oct 31 00:50:16.107841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.105240 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:16.184233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.184187 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:16.382414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.382245 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:50:16.412143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.412117 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 00:50:16.412384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.412370 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 00:50:16.427453 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.427421 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p"] Oct 31 00:50:16.427977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.427958 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 00:50:16.867266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:16.867230 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" event={"ID":"913d0944-09fa-4e45-8373-ebbe5f0f1bca","Type":"ContainerStarted","Data":"21b6e84d394641395d678c11f53d412d73693b8b184fc1b26b58b6d759edb4a7"} Oct 31 00:50:17.876297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:17.876260 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" event={"ID":"913d0944-09fa-4e45-8373-ebbe5f0f1bca","Type":"ContainerStarted","Data":"1ec09b5713d275f947bf2932ace7a287fda28f004bb99e5ca3ecf9c9ac21d6e4"} Oct 31 00:50:17.876851 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:17.876338 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:17.995005 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:17.994924 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" podStartSLOduration=4.006914252 podStartE2EDuration="4.994902395s" totalImagesPullingTime="987.988143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:17.994051285 +0000 UTC m=+1310.196301723" watchObservedRunningTime="2026-10-31 00:50:17.994902395 +0000 UTC m=+1310.197152829" Oct 31 00:50:23.890429 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:23.890394 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:50:25.742851 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.742813 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks"] Oct 31 00:50:25.747894 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.747865 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:25.807699 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.806801 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks"] Oct 31 00:50:25.828477 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.828434 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wb9\" (UniqueName: \"kubernetes.io/projected/d705d3ea-4445-470d-9c2c-0f694e521044-kube-api-access-27wb9\") pod \"payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:25.828683 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.828641 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d705d3ea-4445-470d-9c2c-0f694e521044-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:25.845677 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.845649 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8c2357\"" Oct 31 00:50:25.845677 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.845670 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8c2357-dockercfg-tswj7\"" Oct 31 00:50:25.930477 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.930437 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wb9\" (UniqueName: \"kubernetes.io/projected/d705d3ea-4445-470d-9c2c-0f694e521044-kube-api-access-27wb9\") pod \"payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:25.930644 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.930603 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d705d3ea-4445-470d-9c2c-0f694e521044-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:25.931337 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.931313 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d705d3ea-4445-470d-9c2c-0f694e521044-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:25.990103 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:25.990069 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wb9\" (UniqueName: \"kubernetes.io/projected/d705d3ea-4445-470d-9c2c-0f694e521044-kube-api-access-27wb9\") pod \"payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:26.064051 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:26.063973 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:26.439779 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:26.439748 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks"] Oct 31 00:50:26.440455 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:50:26.440429 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd705d3ea_4445_470d_9c2c_0f694e521044.slice/crio-af4e9304ea31639a352d051695b1e5ddd189e4fcd7183d9516f254fc5306e2c8 WatchSource:0}: Error finding container af4e9304ea31639a352d051695b1e5ddd189e4fcd7183d9516f254fc5306e2c8: Status 404 returned error can't find the container with id af4e9304ea31639a352d051695b1e5ddd189e4fcd7183d9516f254fc5306e2c8 Oct 31 00:50:26.953449 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:26.953403 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" event={"ID":"d705d3ea-4445-470d-9c2c-0f694e521044","Type":"ContainerStarted","Data":"92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449"} Oct 31 00:50:26.953877 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:26.953580 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:26.953877 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:26.953625 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" event={"ID":"d705d3ea-4445-470d-9c2c-0f694e521044","Type":"ContainerStarted","Data":"af4e9304ea31639a352d051695b1e5ddd189e4fcd7183d9516f254fc5306e2c8"} Oct 31 00:50:27.068902 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:27.068846 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" podStartSLOduration=2.0688335 podStartE2EDuration="2.0688335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:27.065448014 +0000 UTC m=+1319.267698447" watchObservedRunningTime="2026-10-31 00:50:27.0688335 +0000 UTC m=+1319.271083933" Oct 31 00:50:37.021870 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:50:37.021824 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:50:37.964165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:37.964130 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:50:43.551808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.551419 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6"] Oct 31 00:50:43.565084 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.564085 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.608165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.608098 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6"] Oct 31 00:50:43.644910 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.644879 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"distinct-2-w1-b63c24-kserve-self-signed-certs\"" Oct 31 00:50:43.655272 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.655239 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tn8h\" (UniqueName: \"kubernetes.io/projected/4e45c970-b6d8-48e0-b780-ec5444252491-kube-api-access-7tn8h\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.655460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.655300 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-home\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.655460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.655440 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-model-cache\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.655597 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.655488 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e45c970-b6d8-48e0-b780-ec5444252491-tls-certs\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.655671 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.655644 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-tmp-dir\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.655729 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.655711 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-dshm\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.756725 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.756684 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tn8h\" (UniqueName: \"kubernetes.io/projected/4e45c970-b6d8-48e0-b780-ec5444252491-kube-api-access-7tn8h\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.756922 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.756764 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-home\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.756981 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.756952 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-model-cache\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.757042 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.757025 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e45c970-b6d8-48e0-b780-ec5444252491-tls-certs\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.757141 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.757120 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-home\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.757141 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.757132 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-tmp-dir\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.757259 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.757190 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-dshm\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.757362 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.757342 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-model-cache\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.757463 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.757443 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-tmp-dir\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.759433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.759406 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-dshm\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.759587 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.759566 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e45c970-b6d8-48e0-b780-ec5444252491-tls-certs\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.815860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.815749 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tn8h\" (UniqueName: \"kubernetes.io/projected/4e45c970-b6d8-48e0-b780-ec5444252491-kube-api-access-7tn8h\") pod \"distinct-2-w1-b63c24-kserve-77548c5479-b2jq6\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:43.882809 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:43.882760 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:44.376739 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:44.376698 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6"] Oct 31 00:50:44.382455 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:50:44.382407 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e45c970_b6d8_48e0_b780_ec5444252491.slice/crio-ac4ea998480539596a8dd9e414335ce62ec3a8d65d4be76e55d60f9a4e71f687 WatchSource:0}: Error finding container ac4ea998480539596a8dd9e414335ce62ec3a8d65d4be76e55d60f9a4e71f687: Status 404 returned error can't find the container with id ac4ea998480539596a8dd9e414335ce62ec3a8d65d4be76e55d60f9a4e71f687 Oct 31 00:50:45.077291 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.077252 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9"] Oct 31 00:50:45.081683 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.081659 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.088256 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.088231 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" event={"ID":"4e45c970-b6d8-48e0-b780-ec5444252491","Type":"ContainerStarted","Data":"2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7"} Oct 31 00:50:45.088399 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.088266 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" event={"ID":"4e45c970-b6d8-48e0-b780-ec5444252491","Type":"ContainerStarted","Data":"ac4ea998480539596a8dd9e414335ce62ec3a8d65d4be76e55d60f9a4e71f687"} Oct 31 00:50:45.132939 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.132910 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b63c24-openshift-default-dockercfg-58m59\"" Oct 31 00:50:45.183926 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.183894 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184098 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.183963 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184098 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.184000 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184098 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.184058 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184098 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.184080 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.184195 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.184238 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.184255 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.184324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.184277 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wwc\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-kube-api-access-p9wwc\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.186076 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.186048 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9"] Oct 31 00:50:45.285141 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285109 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wwc\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-kube-api-access-p9wwc\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285310 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285224 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285310 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285259 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285310 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285302 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285372 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285459 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285396 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285565 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285533 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285583 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285606 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285774 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285751 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285882 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.285905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.285896 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.286221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.286200 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.286221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.286208 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.287955 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.287932 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.288167 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.288150 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.312587 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.312542 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" podStartSLOduration=2.31252874 podStartE2EDuration="2.31252874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:45.31007361 +0000 UTC m=+1337.512324043" watchObservedRunningTime="2026-10-31 00:50:45.31252874 +0000 UTC m=+1337.514779173" Oct 31 00:50:45.346132 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.346065 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.346266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.346194 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wwc\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-kube-api-access-p9wwc\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.397220 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.397188 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:45.845213 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.845181 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9"] Oct 31 00:50:45.845658 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:50:45.845632 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73c23d4_bcc2_4b35_9d1d_4868db6698f5.slice/crio-94b831a8b6144d5c06e30418ebafcf339b6cddbbb15a1a943c2a8d1d1427458d WatchSource:0}: Error finding container 94b831a8b6144d5c06e30418ebafcf339b6cddbbb15a1a943c2a8d1d1427458d: Status 404 returned error can't find the container with id 94b831a8b6144d5c06e30418ebafcf339b6cddbbb15a1a943c2a8d1d1427458d Oct 31 00:50:45.848351 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.848315 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:45.848425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.848382 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:45.848425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.848411 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:45.883217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:45.883194 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:46.097161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:46.097078 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" event={"ID":"f73c23d4-bcc2-4b35-9d1d-4868db6698f5","Type":"ContainerStarted","Data":"fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772"} Oct 31 00:50:46.097161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:46.097136 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" event={"ID":"f73c23d4-bcc2-4b35-9d1d-4868db6698f5","Type":"ContainerStarted","Data":"94b831a8b6144d5c06e30418ebafcf339b6cddbbb15a1a943c2a8d1d1427458d"} Oct 31 00:50:46.217368 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:46.217314 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podStartSLOduration=1.217296574 podStartE2EDuration="1.217296574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:46.21312631 +0000 UTC m=+1338.415376742" watchObservedRunningTime="2026-10-31 00:50:46.217296574 +0000 UTC m=+1338.419547007" Oct 31 00:50:46.398281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:46.398185 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:46.399770 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:46.399736 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:50:46.399911 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:46.399800 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:50:47.398480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:47.398442 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:50:47.398941 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:47.398499 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:50:48.398067 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:48.398025 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:50:48.398249 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:48.398083 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:50:49.398300 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:49.398263 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:50:49.398715 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:49.398318 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:50:50.398332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:50.398297 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:50:50.398737 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:50.398352 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:50:51.398462 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:51.398417 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:50:51.398874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:51.398510 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:50:52.397803 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:52.397756 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:50:52.398010 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:52.397891 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 00:50:53.411140 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:53.411100 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:53.412625 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:53.412579 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:53.418058 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:53.418033 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 00:50:53.883049 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:53.883011 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:53.897267 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:53.897234 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:50:54.169732 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:50:54.169653 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 00:51:07.018919 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:51:07.018874 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:51:22.546193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.546103 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4"] Oct 31 00:51:22.551818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.551777 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.606302 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.606263 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4"] Oct 31 00:51:22.647595 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.647562 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7e6fe88b\"/\"openshift-service-ca.crt\"" Oct 31 00:51:22.648393 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.648368 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7e6fe88b\"/\"kube-root-ca.crt\"" Oct 31 00:51:22.651004 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.650974 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7e6fe88b\"/\"default-dockercfg-krwjx\"" Oct 31 00:51:22.651004 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.650974 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7e6fe88b\"/\"e2e-lifecycle-model-7e6fe88b-kserve-self-signed-certs\"" Oct 31 00:51:22.652683 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.652660 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538a73cd-c247-4e5b-bd78-f1963689bb8b-tls-certs\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.652764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.652720 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-model-cache\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.652836 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.652815 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-dshm\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.652875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.652843 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxrg\" (UniqueName: \"kubernetes.io/projected/538a73cd-c247-4e5b-bd78-f1963689bb8b-kube-api-access-htxrg\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.653000 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.652982 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-home\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.653061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.653015 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-tmp-dir\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.754796 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.754754 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538a73cd-c247-4e5b-bd78-f1963689bb8b-tls-certs\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.754843 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-model-cache\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.754888 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-dshm\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.755023 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxrg\" (UniqueName: \"kubernetes.io/projected/538a73cd-c247-4e5b-bd78-f1963689bb8b-kube-api-access-htxrg\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755210 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.755149 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-home\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755210 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.755180 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-tmp-dir\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755539 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.755507 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-model-cache\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755539 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.755525 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-tmp-dir\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.755850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.755830 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-home\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.757468 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.757442 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-dshm\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.758183 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.758161 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538a73cd-c247-4e5b-bd78-f1963689bb8b-tls-certs\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.814163 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.814093 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxrg\" (UniqueName: \"kubernetes.io/projected/538a73cd-c247-4e5b-bd78-f1963689bb8b-kube-api-access-htxrg\") pod \"e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:22.866830 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:22.866773 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:23.248324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:23.248295 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4"] Oct 31 00:51:23.248837 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:51:23.248809 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod538a73cd_c247_4e5b_bd78_f1963689bb8b.slice/crio-cf0ea44fa9328db1e8441ee8673bbbf7cb985f92e7f106ae73a03b4153d080d4 WatchSource:0}: Error finding container cf0ea44fa9328db1e8441ee8673bbbf7cb985f92e7f106ae73a03b4153d080d4: Status 404 returned error can't find the container with id cf0ea44fa9328db1e8441ee8673bbbf7cb985f92e7f106ae73a03b4153d080d4 Oct 31 00:51:23.403353 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:23.403317 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" event={"ID":"538a73cd-c247-4e5b-bd78-f1963689bb8b","Type":"ContainerStarted","Data":"6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc"} Oct 31 00:51:23.403534 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:23.403362 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" event={"ID":"538a73cd-c247-4e5b-bd78-f1963689bb8b","Type":"ContainerStarted","Data":"cf0ea44fa9328db1e8441ee8673bbbf7cb985f92e7f106ae73a03b4153d080d4"} Oct 31 00:51:23.550880 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:23.538501 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" podStartSLOduration=1.538479636 podStartE2EDuration="1.538479636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:23.537219783 +0000 UTC m=+1375.739470216" watchObservedRunningTime="2026-10-31 00:51:23.538479636 +0000 UTC m=+1375.740730070" Oct 31 00:51:23.867359 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:23.867263 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:32.867469 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:32.867433 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:32.881363 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:32.881334 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:33.492307 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:33.492276 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:51:37.020614 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:51:37.020500 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:51:47.025017 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:47.024973 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:51:47.025693 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:47.025657 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:51:50.119343 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.119307 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c"] Oct 31 00:51:50.123515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.123493 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175379 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-credential-socket\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175425 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-envoy\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175461 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istiod-ca-cert\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175516 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-certs\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175543 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-podinfo\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175591 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwdf\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-kube-api-access-2rwdf\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175676 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-socket\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175708 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-token\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.175734 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-data\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.181340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.177623 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3c636f-openshift-default-dockercfg-hhw5z\"" Oct 31 00:51:50.231146 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.231109 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c"] Oct 31 00:51:50.278154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278093 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-credential-socket\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278154 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278140 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-envoy\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278179 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istiod-ca-cert\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278245 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-certs\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278273 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-podinfo\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278328 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwdf\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-kube-api-access-2rwdf\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278424 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-socket\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278463 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-token\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.278511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.278492 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-data\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.279641 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.279314 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-data\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.279641 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.279597 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-credential-socket\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.283196 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.282886 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-certs\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.285230 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.283542 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-socket\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.288752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.288729 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istiod-ca-cert\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.289470 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.289449 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-envoy\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.321561 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.321095 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-podinfo\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.348959 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.348923 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-token\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.351713 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.351674 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwdf\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-kube-api-access-2rwdf\") pod \"e2e-api-b-3c636f-openshift-default-8685f58797-ph54c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.437915 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.437866 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:50.872093 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.872065 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c"] Oct 31 00:51:50.873856 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:51:50.873825 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0b0377_cdf7_4f36_9af5_d5772a1dba9c.slice/crio-67491d3fada784830500d05d9c61f3263b304212b582620875ff83e789e9b9db WatchSource:0}: Error finding container 67491d3fada784830500d05d9c61f3263b304212b582620875ff83e789e9b9db: Status 404 returned error can't find the container with id 67491d3fada784830500d05d9c61f3263b304212b582620875ff83e789e9b9db Oct 31 00:51:50.876526 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.876494 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:51:50.876593 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.876558 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:51:50.876593 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:50.876588 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:51:51.654573 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:51.654537 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" event={"ID":"de0b0377-cdf7-4f36-9af5-d5772a1dba9c","Type":"ContainerStarted","Data":"bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264"} Oct 31 00:51:51.655105 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:51.654588 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" event={"ID":"de0b0377-cdf7-4f36-9af5-d5772a1dba9c","Type":"ContainerStarted","Data":"67491d3fada784830500d05d9c61f3263b304212b582620875ff83e789e9b9db"} Oct 31 00:51:51.819508 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:51.819406 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podStartSLOduration=1.8193924510000001 podStartE2EDuration="1.819392451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:51.817684572 +0000 UTC m=+1404.019935007" watchObservedRunningTime="2026-10-31 00:51:51.819392451 +0000 UTC m=+1404.021642883" Oct 31 00:51:52.438856 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:52.438823 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:52.444780 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:52.444752 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:52.661748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:52.661718 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:51:52.662524 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:51:52.662505 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:52:07.018674 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:52:07.018633 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:52:34.014980 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.014946 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r"] Oct 31 00:52:34.019490 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.019471 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.118394 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.118355 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-69a4bc-serving-cert\"" Oct 31 00:52:34.122153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.122125 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r"] Oct 31 00:52:34.127940 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.127916 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f6493a-ab3c-42bc-845d-daed532cb047-openshift-service-ca\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.128063 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.127978 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jj29\" (UniqueName: \"kubernetes.io/projected/70f6493a-ab3c-42bc-845d-daed532cb047-kube-api-access-4jj29\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.128112 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.128084 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.128160 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.128147 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-metrics-certs\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.229449 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.229411 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f6493a-ab3c-42bc-845d-daed532cb047-openshift-service-ca\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.229632 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.229485 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jj29\" (UniqueName: \"kubernetes.io/projected/70f6493a-ab3c-42bc-845d-daed532cb047-kube-api-access-4jj29\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.229632 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.229533 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.229764 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:52:34.229647 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-serving-cert: secret "maas-api-e2e-api-a-69a4bc-serving-cert" not found Oct 31 00:52:34.229764 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:52:34.229704 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls podName:70f6493a-ab3c-42bc-845d-daed532cb047 nodeName:}" failed. No retries permitted until 2026-10-31 00:52:34.729683414 +0000 UTC m=+1446.931933845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls") pod "maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" (UID: "70f6493a-ab3c-42bc-845d-daed532cb047") : secret "maas-api-e2e-api-a-69a4bc-serving-cert" not found Oct 31 00:52:34.229764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.229718 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-metrics-certs\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.230073 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.230051 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f6493a-ab3c-42bc-845d-daed532cb047-openshift-service-ca\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.232198 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.232172 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-metrics-certs\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.292824 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.292726 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jj29\" (UniqueName: \"kubernetes.io/projected/70f6493a-ab3c-42bc-845d-daed532cb047-kube-api-access-4jj29\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.736586 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.736538 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.738990 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.738966 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls\") pod \"maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:34.933461 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:34.933424 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:35.318237 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:35.318204 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r"] Oct 31 00:52:35.318777 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:52:35.318749 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f6493a_ab3c_42bc_845d_daed532cb047.slice/crio-3ef822e70c89dd150859542668d825057ce94c6b51d60adbd24ae1bcd90a4cf0 WatchSource:0}: Error finding container 3ef822e70c89dd150859542668d825057ce94c6b51d60adbd24ae1bcd90a4cf0: Status 404 returned error can't find the container with id 3ef822e70c89dd150859542668d825057ce94c6b51d60adbd24ae1bcd90a4cf0 Oct 31 00:52:36.063859 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:36.063814 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" event={"ID":"70f6493a-ab3c-42bc-845d-daed532cb047","Type":"ContainerStarted","Data":"3ef822e70c89dd150859542668d825057ce94c6b51d60adbd24ae1bcd90a4cf0"} Oct 31 00:52:37.037223 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:52:37.037181 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:52:37.081673 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:37.081502 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" event={"ID":"70f6493a-ab3c-42bc-845d-daed532cb047","Type":"ContainerStarted","Data":"f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163"} Oct 31 00:52:37.089357 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:37.082737 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:52:37.201361 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:37.201295 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" podStartSLOduration=2.887092981 podStartE2EDuration="4.201274957s" totalImagesPullingTime="1.314181976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:37.195481382 +0000 UTC m=+1449.397731815" watchObservedRunningTime="2026-10-31 00:52:37.201274957 +0000 UTC m=+1449.403525393" Oct 31 00:52:44.111080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:52:44.111037 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:53:01.365951 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:01.365916 5890 generic.go:356] "Generic (PLEG): container finished" podID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerID="bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264" exitCode=0 Oct 31 00:53:01.366424 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:01.365991 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" event={"ID":"de0b0377-cdf7-4f36-9af5-d5772a1dba9c","Type":"ContainerDied","Data":"bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264"} Oct 31 00:53:01.366482 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:01.366437 5890 scope.go:122] "RemoveContainer" containerID="bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264" Oct 31 00:53:01.368109 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:01.368077 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:01.368203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:01.368147 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:01.368203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:01.368184 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:02.378322 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:02.378284 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" event={"ID":"de0b0377-cdf7-4f36-9af5-d5772a1dba9c","Type":"ContainerStarted","Data":"77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715"} Oct 31 00:53:02.438484 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:02.438448 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:53:02.438653 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:02.438639 5890 patch_prober.go:29] interesting pod/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:02.438696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:02.438681 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 00:53:03.438564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:03.438526 5890 patch_prober.go:29] interesting pod/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:03.438971 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:03.438596 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 00:53:04.438546 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:04.438508 5890 patch_prober.go:29] interesting pod/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:04.438749 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:04.438561 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 00:53:05.438888 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:05.438848 5890 patch_prober.go:29] interesting pod/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:05.439298 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:05.438922 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 00:53:06.438657 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:06.438622 5890 patch_prober.go:29] interesting pod/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:06.438876 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:06.438684 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 00:53:07.020418 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:53:07.020378 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:53:07.438950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:07.438909 5890 patch_prober.go:29] interesting pod/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:07.439142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:07.439017 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 00:53:07.662369 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:07.662326 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:53:08.447345 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:08.447312 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:53:08.449281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:08.449254 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:53:09.152416 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.152383 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7289-997b79"] Oct 31 00:53:09.155722 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.155698 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:09.205997 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.205965 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8gxx2\"" Oct 31 00:53:09.207490 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.207465 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-997b79"] Oct 31 00:53:09.318630 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.318595 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwsh\" (UniqueName: \"kubernetes.io/projected/e06032cc-e4bd-4434-8f57-b5692a0e309b-kube-api-access-dvwsh\") pod \"test-curl-7289-997b79\" (UID: \"e06032cc-e4bd-4434-8f57-b5692a0e309b\") " pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:09.419826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.419695 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwsh\" (UniqueName: \"kubernetes.io/projected/e06032cc-e4bd-4434-8f57-b5692a0e309b-kube-api-access-dvwsh\") pod \"test-curl-7289-997b79\" (UID: \"e06032cc-e4bd-4434-8f57-b5692a0e309b\") " pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:09.480386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.480347 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwsh\" (UniqueName: \"kubernetes.io/projected/e06032cc-e4bd-4434-8f57-b5692a0e309b-kube-api-access-dvwsh\") pod \"test-curl-7289-997b79\" (UID: \"e06032cc-e4bd-4434-8f57-b5692a0e309b\") " pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:09.769572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:09.769538 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:10.144453 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:53:10.144422 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode06032cc_e4bd_4434_8f57_b5692a0e309b.slice/crio-46f43835883fcd09b726ad1555022d6fdff32801578c4d4a00d66a7933aaad03 WatchSource:0}: Error finding container 46f43835883fcd09b726ad1555022d6fdff32801578c4d4a00d66a7933aaad03: Status 404 returned error can't find the container with id 46f43835883fcd09b726ad1555022d6fdff32801578c4d4a00d66a7933aaad03 Oct 31 00:53:10.144581 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:10.144479 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-997b79"] Oct 31 00:53:10.464866 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:10.464825 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-997b79" event={"ID":"e06032cc-e4bd-4434-8f57-b5692a0e309b","Type":"ContainerStarted","Data":"46f43835883fcd09b726ad1555022d6fdff32801578c4d4a00d66a7933aaad03"} Oct 31 00:53:13.495798 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:13.495749 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-997b79" event={"ID":"e06032cc-e4bd-4434-8f57-b5692a0e309b","Type":"ContainerStarted","Data":"26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296"} Oct 31 00:53:13.606468 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:13.606407 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7289-997b79" podStartSLOduration=1.368752545 podStartE2EDuration="4.60639222s" totalImagesPullingTime="3.237639675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:13.604970874 +0000 UTC m=+1485.807221330" watchObservedRunningTime="2026-10-31 00:53:13.60639222 +0000 UTC m=+1485.808642652" Oct 31 00:53:15.107119 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:15.107082 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7289-997b79"] Oct 31 00:53:15.111031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:15.110996 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7289-997b79"] Oct 31 00:53:15.509403 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:15.509361 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7289-997b79" podUID="e06032cc-e4bd-4434-8f57-b5692a0e309b" containerName="test-curl-7289-997b79" containerID="cri-o://26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296" gracePeriod=2 Oct 31 00:53:15.558461 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:53:15.558418 5890 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-997b79\" is forbidden: User \"system:node:ip-10-0-143-105.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-105.us-west-2.compute.internal' and this object" podUID="e06032cc-e4bd-4434-8f57-b5692a0e309b" pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:17.766492 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:17.766432 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:17.852710 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:17.852676 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06032cc-e4bd-4434-8f57-b5692a0e309b-kube-api-access-dvwsh\" (UniqueName: \"kubernetes.io/projected/e06032cc-e4bd-4434-8f57-b5692a0e309b-kube-api-access-dvwsh\") pod \"e06032cc-e4bd-4434-8f57-b5692a0e309b\" (UID: \"e06032cc-e4bd-4434-8f57-b5692a0e309b\") " Oct 31 00:53:17.855111 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:17.855084 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06032cc-e4bd-4434-8f57-b5692a0e309b-kube-api-access-dvwsh" pod "e06032cc-e4bd-4434-8f57-b5692a0e309b" (UID: "e06032cc-e4bd-4434-8f57-b5692a0e309b"). InnerVolumeSpecName "kube-api-access-dvwsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:17.934824 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:17.934762 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06032cc-e4bd-4434-8f57-b5692a0e309b" path="/var/lib/kubelet/pods/e06032cc-e4bd-4434-8f57-b5692a0e309b/volumes" Oct 31 00:53:17.954128 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:17.954090 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwsh\" (UniqueName: \"kubernetes.io/projected/e06032cc-e4bd-4434-8f57-b5692a0e309b-kube-api-access-dvwsh\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:18.539483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:18.539438 5890 generic.go:356] "Generic (PLEG): container finished" podID="e06032cc-e4bd-4434-8f57-b5692a0e309b" containerID="26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296" exitCode=137 Oct 31 00:53:18.539678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:18.539490 5890 scope.go:122] "RemoveContainer" containerID="26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296" Oct 31 00:53:18.539678 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:18.539524 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-997b79" Oct 31 00:53:18.556362 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:18.552922 5890 scope.go:122] "RemoveContainer" containerID="26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296" Oct 31 00:53:18.556362 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:53:18.553268 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296\": container with ID starting with 26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296 not found: ID does not exist" containerID="26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296" Oct 31 00:53:18.556362 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:18.553304 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296"} err="failed to get container status \"26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296\": rpc error: code = NotFound desc = could not find container \"26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296\": container with ID starting with 26e0fc87425419daea7b4149d52a87fd4393c18e93ad16681ee684be10d8a296 not found: ID does not exist" Oct 31 00:53:27.547065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:27.547024 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb"] Oct 31 00:53:28.661714 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:28.661680 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:53:28.776748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:28.776718 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:53:32.391879 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.391839 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f"] Oct 31 00:53:32.392498 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.392484 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e06032cc-e4bd-4434-8f57-b5692a0e309b" containerName="test-curl-7289-997b79" Oct 31 00:53:32.392542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.392498 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06032cc-e4bd-4434-8f57-b5692a0e309b" containerName="test-curl-7289-997b79" Oct 31 00:53:32.392631 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.392622 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06032cc-e4bd-4434-8f57-b5692a0e309b" containerName="test-curl-7289-997b79" Oct 31 00:53:32.397685 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.397664 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.483205 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.483176 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e942ab-openshift-default-dockercfg-phb2n\"" Oct 31 00:53:32.511991 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.511949 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-certs\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512104 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-envoy\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512201 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-credential-socket\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512338 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512322 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-token\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512374 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512359 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-socket\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512407 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512385 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istiod-ca-cert\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512506 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512490 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd55p\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-kube-api-access-zd55p\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512557 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512543 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-podinfo\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.512636 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.512622 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-data\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.552839 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.552769 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" podUID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" containerName="payload-pre-processing" containerID="cri-o://4940e62556d41c7db9a5c20d67235e2415a80346d8574936f521b00f5534dc56" gracePeriod=25 Oct 31 00:53:32.614950 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.614890 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f"] Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.633157 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-certs\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.636944 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-envoy\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637014 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-credential-socket\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637144 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-token\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637180 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-socket\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637203 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istiod-ca-cert\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637316 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd55p\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-kube-api-access-zd55p\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637368 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-podinfo\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637447 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-data\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.637996 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-data\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.640806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.638259 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-certs\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.661479 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.652500 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-socket\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.661479 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.652776 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-credential-socket\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.671030 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.670989 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istiod-ca-cert\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.675233 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.675170 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-envoy\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.676826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.676753 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-podinfo\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.721140 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.721070 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-token\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:32.722481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:32.722450 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd55p\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-kube-api-access-zd55p\") pod \"e2e-api-b-e942ab-openshift-default-9c697d45-s727f\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:33.011851 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.011810 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:33.535965 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.535927 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f"] Oct 31 00:53:33.536371 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:53:33.536186 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d6e5c8_761b_46c9_9ab3_eefd2844d806.slice/crio-df37b30541675a518a4e7276bfe651de7545ac55a4d1ee124d15ece331d8104f WatchSource:0}: Error finding container df37b30541675a518a4e7276bfe651de7545ac55a4d1ee124d15ece331d8104f: Status 404 returned error can't find the container with id df37b30541675a518a4e7276bfe651de7545ac55a4d1ee124d15ece331d8104f Oct 31 00:53:33.538737 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.538705 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:33.538801 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.538771 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:33.538841 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.538815 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:33.751941 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.751899 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" event={"ID":"73d6e5c8-761b-46c9-9ab3-eefd2844d806","Type":"ContainerStarted","Data":"f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381"} Oct 31 00:53:33.752126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.751964 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" event={"ID":"73d6e5c8-761b-46c9-9ab3-eefd2844d806","Type":"ContainerStarted","Data":"df37b30541675a518a4e7276bfe651de7545ac55a4d1ee124d15ece331d8104f"} Oct 31 00:53:33.870432 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:33.870339 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" podStartSLOduration=1.8703249990000002 podStartE2EDuration="1.870324999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:33.867464733 +0000 UTC m=+1506.069715168" watchObservedRunningTime="2026-10-31 00:53:33.870324999 +0000 UTC m=+1506.072575432" Oct 31 00:53:34.012699 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:34.012662 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:34.018472 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:34.018444 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:34.759876 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:34.759846 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:34.760886 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:34.760857 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:53:35.915287 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:35.915251 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" podUID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:53:37.019672 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:53:37.019635 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:53:45.915476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:45.915441 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" podUID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:53:51.232344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:51.232297 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks"] Oct 31 00:53:52.983176 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:52.983133 5890 generic.go:356] "Generic (PLEG): container finished" podID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" containerID="4940e62556d41c7db9a5c20d67235e2415a80346d8574936f521b00f5534dc56" exitCode=0 Oct 31 00:53:52.983808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:52.983769 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" event={"ID":"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7","Type":"ContainerDied","Data":"4940e62556d41c7db9a5c20d67235e2415a80346d8574936f521b00f5534dc56"} Oct 31 00:53:53.035352 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.035321 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:53:53.170856 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.170822 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-kube-api-access-mftkh\" (UniqueName: \"kubernetes.io/projected/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-kube-api-access-mftkh\") pod \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " Oct 31 00:53:53.171061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.170985 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-plugins-config-volume\") pod \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\" (UID: \"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7\") " Oct 31 00:53:53.171721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.171687 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-plugins-config-volume" pod "f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" (UID: "f99c6dd3-2417-44c0-b12c-9a5ea790b6e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:53.176968 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.176935 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-kube-api-access-mftkh" pod "f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" (UID: "f99c6dd3-2417-44c0-b12c-9a5ea790b6e7"). InnerVolumeSpecName "kube-api-access-mftkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:53.273417 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.273338 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:53.273417 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.273372 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftkh\" (UniqueName: \"kubernetes.io/projected/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7-kube-api-access-mftkh\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:53.993123 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.993096 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" Oct 31 00:53:53.993596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.993093 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb" event={"ID":"f99c6dd3-2417-44c0-b12c-9a5ea790b6e7","Type":"ContainerDied","Data":"56e2e764358b3e3257e743ded853393398d6ff43cd64ea4783d322fc57536a08"} Oct 31 00:53:53.993596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:53.993229 5890 scope.go:122] "RemoveContainer" containerID="4940e62556d41c7db9a5c20d67235e2415a80346d8574936f521b00f5534dc56" Oct 31 00:53:54.153694 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:54.153650 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb"] Oct 31 00:53:54.199605 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:54.199525 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-sqxmb"] Oct 31 00:53:55.935203 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:55.935163 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" path="/var/lib/kubelet/pods/f99c6dd3-2417-44c0-b12c-9a5ea790b6e7/volumes" Oct 31 00:53:56.233612 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.233511 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" podUID="d705d3ea-4445-470d-9c2c-0f694e521044" containerName="payload-pre-processing" containerID="cri-o://92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449" gracePeriod=25 Oct 31 00:53:56.548231 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.548205 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:53:56.739388 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.739354 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d705d3ea-4445-470d-9c2c-0f694e521044-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d705d3ea-4445-470d-9c2c-0f694e521044-plugins-config-volume\") pod \"d705d3ea-4445-470d-9c2c-0f694e521044\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " Oct 31 00:53:56.739549 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.739415 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d705d3ea-4445-470d-9c2c-0f694e521044-kube-api-access-27wb9\" (UniqueName: \"kubernetes.io/projected/d705d3ea-4445-470d-9c2c-0f694e521044-kube-api-access-27wb9\") pod \"d705d3ea-4445-470d-9c2c-0f694e521044\" (UID: \"d705d3ea-4445-470d-9c2c-0f694e521044\") " Oct 31 00:53:56.739891 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.739857 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d705d3ea-4445-470d-9c2c-0f694e521044-plugins-config-volume" pod "d705d3ea-4445-470d-9c2c-0f694e521044" (UID: "d705d3ea-4445-470d-9c2c-0f694e521044"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:56.740224 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.740200 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d705d3ea-4445-470d-9c2c-0f694e521044-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:56.741651 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.741625 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d705d3ea-4445-470d-9c2c-0f694e521044-kube-api-access-27wb9" pod "d705d3ea-4445-470d-9c2c-0f694e521044" (UID: "d705d3ea-4445-470d-9c2c-0f694e521044"). InnerVolumeSpecName "kube-api-access-27wb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:56.842050 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:56.841964 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27wb9\" (UniqueName: \"kubernetes.io/projected/d705d3ea-4445-470d-9c2c-0f694e521044-kube-api-access-27wb9\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:57.026805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.023909 5890 generic.go:356] "Generic (PLEG): container finished" podID="d705d3ea-4445-470d-9c2c-0f694e521044" containerID="92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449" exitCode=0 Oct 31 00:53:57.026805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.024105 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" event={"ID":"d705d3ea-4445-470d-9c2c-0f694e521044","Type":"ContainerDied","Data":"92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449"} Oct 31 00:53:57.026805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.024156 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" event={"ID":"d705d3ea-4445-470d-9c2c-0f694e521044","Type":"ContainerDied","Data":"af4e9304ea31639a352d051695b1e5ddd189e4fcd7183d9516f254fc5306e2c8"} Oct 31 00:53:57.026805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.024183 5890 scope.go:122] "RemoveContainer" containerID="92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449" Oct 31 00:53:57.026805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.024332 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks" Oct 31 00:53:57.107941 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.107916 5890 scope.go:122] "RemoveContainer" containerID="92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449" Oct 31 00:53:57.111803 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:53:57.108622 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449\": container with ID starting with 92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449 not found: ID does not exist" containerID="92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449" Oct 31 00:53:57.111803 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.108706 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449"} err="failed to get container status \"92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449\": rpc error: code = NotFound desc = could not find container \"92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449\": container with ID starting with 92481d83ddac75e89f596bf362820011b7682d37576d78e9ce90d13a88f4e449 not found: ID does not exist" Oct 31 00:53:57.292949 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.292912 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks"] Oct 31 00:53:57.329842 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.310878 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8c2357-bb9744f4c-jxkks"] Oct 31 00:53:57.937610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:53:57.937568 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d705d3ea-4445-470d-9c2c-0f694e521044" path="/var/lib/kubelet/pods/d705d3ea-4445-470d-9c2c-0f694e521044/volumes" Oct 31 00:54:07.020098 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:54:07.020061 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:54:29.972075 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.972039 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm"] Oct 31 00:54:29.972736 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.972717 5890 cpu_manager.go:414] "containerMap: removing container" podUID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" containerName="payload-pre-processing" Oct 31 00:54:29.972844 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.972736 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" containerName="payload-pre-processing" Oct 31 00:54:29.972844 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.972748 5890 cpu_manager.go:414] "containerMap: removing container" podUID="d705d3ea-4445-470d-9c2c-0f694e521044" containerName="payload-pre-processing" Oct 31 00:54:29.972844 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.972754 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d705d3ea-4445-470d-9c2c-0f694e521044" containerName="payload-pre-processing" Oct 31 00:54:29.972996 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.972883 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99c6dd3-2417-44c0-b12c-9a5ea790b6e7" containerName="payload-pre-processing" Oct 31 00:54:29.972996 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.972895 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="d705d3ea-4445-470d-9c2c-0f694e521044" containerName="payload-pre-processing" Oct 31 00:54:29.978224 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:29.978198 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.046857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.046556 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ac952a-openshift-default-dockercfg-n49m2\"" Oct 31 00:54:30.105516 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.105484 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm"] Oct 31 00:54:30.174416 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174377 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-token\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174463 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-envoy\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174492 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-credential-socket\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174518 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-podinfo\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174546 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fplf2\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-kube-api-access-fplf2\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174690 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b373e-2051-4517-b1c0-a25c5f61d217-istiod-ca-cert\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174736 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-certs\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174778 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-socket\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.174907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.174860 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-data\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.276975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.276874 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b373e-2051-4517-b1c0-a25c5f61d217-istiod-ca-cert\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.276975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.276948 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-certs\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.276992 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-socket\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.277056 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-data\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.277104 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-token\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.277182 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-envoy\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.277210 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-credential-socket\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.277234 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-podinfo\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277270 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.277262 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fplf2\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-kube-api-access-fplf2\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.277712 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.277685 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b373e-2051-4517-b1c0-a25c5f61d217-istiod-ca-cert\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.278126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.278095 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-certs\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.278266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.278176 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-credential-socket\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.278266 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.278254 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-socket\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.278503 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.278484 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-data\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.283826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.282115 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-envoy\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.284099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.284077 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-podinfo\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.338873 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.338841 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-token\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.339136 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.339111 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fplf2\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-kube-api-access-fplf2\") pod \"e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:30.597881 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:30.597780 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:31.024961 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.024923 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm"] Oct 31 00:54:31.025369 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:54:31.024920 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64b373e_2051_4517_b1c0_a25c5f61d217.slice/crio-423f1b23056767b30f6f5643cbd797a2c0b7639b00dd291f83039a11513fd833 WatchSource:0}: Error finding container 423f1b23056767b30f6f5643cbd797a2c0b7639b00dd291f83039a11513fd833: Status 404 returned error can't find the container with id 423f1b23056767b30f6f5643cbd797a2c0b7639b00dd291f83039a11513fd833 Oct 31 00:54:31.027689 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.027658 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:31.027758 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.027741 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:31.027833 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.027771 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:31.318729 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.318641 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" event={"ID":"b64b373e-2051-4517-b1c0-a25c5f61d217","Type":"ContainerStarted","Data":"52cc85aae700c11dc46b80e04f7ce29e6a96151b99525447bf4a1bcf655c59a7"} Oct 31 00:54:31.318913 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.318740 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" event={"ID":"b64b373e-2051-4517-b1c0-a25c5f61d217","Type":"ContainerStarted","Data":"423f1b23056767b30f6f5643cbd797a2c0b7639b00dd291f83039a11513fd833"} Oct 31 00:54:31.436049 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.435994 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" podStartSLOduration=2.435981162 podStartE2EDuration="2.435981162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:31.434215154 +0000 UTC m=+1563.636465585" watchObservedRunningTime="2026-10-31 00:54:31.435981162 +0000 UTC m=+1563.638231595" Oct 31 00:54:31.598418 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:31.598306 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:32.604157 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:32.604125 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:33.340768 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:33.340731 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:33.341688 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:33.341666 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:54:37.021750 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:54:37.021705 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:54:38.028419 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.028381 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p"] Oct 31 00:54:38.029405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.029364 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" podUID="913d0944-09fa-4e45-8373-ebbe5f0f1bca" containerName="maas-api" containerID="cri-o://1ec09b5713d275f947bf2932ace7a287fda28f004bb99e5ca3ecf9c9ac21d6e4" gracePeriod=30 Oct 31 00:54:38.508826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.505140 5890 generic.go:356] "Generic (PLEG): container finished" podID="913d0944-09fa-4e45-8373-ebbe5f0f1bca" containerID="1ec09b5713d275f947bf2932ace7a287fda28f004bb99e5ca3ecf9c9ac21d6e4" exitCode=0 Oct 31 00:54:38.508826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.505294 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" event={"ID":"913d0944-09fa-4e45-8373-ebbe5f0f1bca","Type":"ContainerDied","Data":"1ec09b5713d275f947bf2932ace7a287fda28f004bb99e5ca3ecf9c9ac21d6e4"} Oct 31 00:54:38.725345 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.725317 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:54:38.818094 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.818061 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-maas-api-tls\") pod \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " Oct 31 00:54:38.818260 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.818103 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-metrics-certs\") pod \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " Oct 31 00:54:38.818260 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.818216 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913d0944-09fa-4e45-8373-ebbe5f0f1bca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913d0944-09fa-4e45-8373-ebbe5f0f1bca-openshift-service-ca\") pod \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " Oct 31 00:54:38.818260 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.818250 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913d0944-09fa-4e45-8373-ebbe5f0f1bca-kube-api-access-rthd9\" (UniqueName: \"kubernetes.io/projected/913d0944-09fa-4e45-8373-ebbe5f0f1bca-kube-api-access-rthd9\") pod \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\" (UID: \"913d0944-09fa-4e45-8373-ebbe5f0f1bca\") " Oct 31 00:54:38.818541 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.818509 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913d0944-09fa-4e45-8373-ebbe5f0f1bca-openshift-service-ca" pod "913d0944-09fa-4e45-8373-ebbe5f0f1bca" (UID: "913d0944-09fa-4e45-8373-ebbe5f0f1bca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:38.819076 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.819054 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913d0944-09fa-4e45-8373-ebbe5f0f1bca-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:38.820420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.820395 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-maas-api-tls" pod "913d0944-09fa-4e45-8373-ebbe5f0f1bca" (UID: "913d0944-09fa-4e45-8373-ebbe5f0f1bca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:38.820420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.820410 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913d0944-09fa-4e45-8373-ebbe5f0f1bca-kube-api-access-rthd9" pod "913d0944-09fa-4e45-8373-ebbe5f0f1bca" (UID: "913d0944-09fa-4e45-8373-ebbe5f0f1bca"). InnerVolumeSpecName "kube-api-access-rthd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:38.820546 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.820446 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-metrics-certs" pod "913d0944-09fa-4e45-8373-ebbe5f0f1bca" (UID: "913d0944-09fa-4e45-8373-ebbe5f0f1bca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:38.920908 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.920871 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rthd9\" (UniqueName: \"kubernetes.io/projected/913d0944-09fa-4e45-8373-ebbe5f0f1bca-kube-api-access-rthd9\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:38.920908 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.920909 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:38.921095 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:38.920925 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913d0944-09fa-4e45-8373-ebbe5f0f1bca-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:39.513745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:39.513696 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" Oct 31 00:54:39.513745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:39.513725 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p" event={"ID":"913d0944-09fa-4e45-8373-ebbe5f0f1bca","Type":"ContainerDied","Data":"21b6e84d394641395d678c11f53d412d73693b8b184fc1b26b58b6d759edb4a7"} Oct 31 00:54:39.514281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:39.513808 5890 scope.go:122] "RemoveContainer" containerID="1ec09b5713d275f947bf2932ace7a287fda28f004bb99e5ca3ecf9c9ac21d6e4" Oct 31 00:54:39.685914 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:39.685873 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p"] Oct 31 00:54:39.689252 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:39.689224 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e6fe88b-55755c67c7-sxr4p"] Oct 31 00:54:39.933978 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:39.933943 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913d0944-09fa-4e45-8373-ebbe5f0f1bca" path="/var/lib/kubelet/pods/913d0944-09fa-4e45-8373-ebbe5f0f1bca/volumes" Oct 31 00:54:46.684627 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:46.684593 5890 generic.go:356] "Generic (PLEG): container finished" podID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerID="f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381" exitCode=0 Oct 31 00:54:46.685073 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:46.684665 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" event={"ID":"73d6e5c8-761b-46c9-9ab3-eefd2844d806","Type":"ContainerDied","Data":"f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381"} Oct 31 00:54:46.685156 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:46.685141 5890 scope.go:122] "RemoveContainer" containerID="f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381" Oct 31 00:54:46.686933 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:46.686897 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:46.687016 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:46.686975 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:46.687053 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:46.687029 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:47.694059 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:47.694024 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" event={"ID":"73d6e5c8-761b-46c9-9ab3-eefd2844d806","Type":"ContainerStarted","Data":"27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa"} Oct 31 00:54:48.012761 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:48.012678 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:54:48.014059 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:48.014030 5890 patch_prober.go:29] interesting pod/e2e-api-b-e942ab-openshift-default-9c697d45-s727f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:54:48.014184 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:48.014081 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 00:54:49.012758 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:49.012722 5890 patch_prober.go:29] interesting pod/e2e-api-b-e942ab-openshift-default-9c697d45-s727f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:54:49.013181 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:49.012774 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 00:54:49.768811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:49.765881 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:54:50.012692 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:50.012658 5890 patch_prober.go:29] interesting pod/e2e-api-b-e942ab-openshift-default-9c697d45-s727f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:54:50.012930 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:50.012719 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 00:54:51.012400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:51.012369 5890 patch_prober.go:29] interesting pod/e2e-api-b-e942ab-openshift-default-9c697d45-s727f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:54:51.012581 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:51.012423 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 00:54:52.015808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:52.014954 5890 patch_prober.go:29] interesting pod/e2e-api-b-e942ab-openshift-default-9c697d45-s727f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:54:52.015808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:52.015026 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 00:54:53.017193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:53.017160 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:54:53.018224 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:53.018200 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:54:55.501818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.495319 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt"] Oct 31 00:54:55.501818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.496348 5890 cpu_manager.go:414] "containerMap: removing container" podUID="913d0944-09fa-4e45-8373-ebbe5f0f1bca" containerName="maas-api" Oct 31 00:54:55.501818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.496369 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913d0944-09fa-4e45-8373-ebbe5f0f1bca" containerName="maas-api" Oct 31 00:54:55.501818 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.496514 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="913d0944-09fa-4e45-8373-ebbe5f0f1bca" containerName="maas-api" Oct 31 00:54:55.507812 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.503981 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.554078 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.554042 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt"] Oct 31 00:54:55.603079 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.603049 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7e6fe88b\"" Oct 31 00:54:55.603881 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.603326 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7e6fe88b-dockercfg-kd4cq\"" Oct 31 00:54:55.623282 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.623247 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrclt\" (UniqueName: \"kubernetes.io/projected/cbfe5036-6f6e-49e3-9332-d558dd54b375-kube-api-access-zrclt\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.623439 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.623375 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfe5036-6f6e-49e3-9332-d558dd54b375-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.647481 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.647450 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2"] Oct 31 00:54:55.650710 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.650688 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:55.724975 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.724936 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8x8\" (UniqueName: \"kubernetes.io/projected/e21029b5-0178-4c15-9013-61e8b08b7f90-kube-api-access-9t8x8\") pod \"payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:55.725305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.725280 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrclt\" (UniqueName: \"kubernetes.io/projected/cbfe5036-6f6e-49e3-9332-d558dd54b375-kube-api-access-zrclt\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.725507 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.725490 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfe5036-6f6e-49e3-9332-d558dd54b375-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.727185 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.725729 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21029b5-0178-4c15-9013-61e8b08b7f90-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:55.728960 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.728923 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfe5036-6f6e-49e3-9332-d558dd54b375-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.734054 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.734034 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2"] Oct 31 00:54:55.788340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.788238 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrclt\" (UniqueName: \"kubernetes.io/projected/cbfe5036-6f6e-49e3-9332-d558dd54b375-kube-api-access-zrclt\") pod \"payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.820104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.820065 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:55.829232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.829196 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8x8\" (UniqueName: \"kubernetes.io/projected/e21029b5-0178-4c15-9013-61e8b08b7f90-kube-api-access-9t8x8\") pod \"payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:55.829433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.829408 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21029b5-0178-4c15-9013-61e8b08b7f90-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:55.830408 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.830379 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21029b5-0178-4c15-9013-61e8b08b7f90-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:55.905649 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:55.905621 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8x8\" (UniqueName: \"kubernetes.io/projected/e21029b5-0178-4c15-9013-61e8b08b7f90-kube-api-access-9t8x8\") pod \"payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:56.043460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.043379 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:56.224120 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.224089 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt"] Oct 31 00:54:56.224686 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:54:56.224662 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfe5036_6f6e_49e3_9332_d558dd54b375.slice/crio-dea10d426c91772f09510726b67cd6f406e79e9466d22d241af49ea7647c27b0 WatchSource:0}: Error finding container dea10d426c91772f09510726b67cd6f406e79e9466d22d241af49ea7647c27b0: Status 404 returned error can't find the container with id dea10d426c91772f09510726b67cd6f406e79e9466d22d241af49ea7647c27b0 Oct 31 00:54:56.423770 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.423732 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2"] Oct 31 00:54:56.424225 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:54:56.423895 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21029b5_0178_4c15_9013_61e8b08b7f90.slice/crio-291477be58c963cb10395df09b5776b1569ed7b71ebcd2802947c00499969d35 WatchSource:0}: Error finding container 291477be58c963cb10395df09b5776b1569ed7b71ebcd2802947c00499969d35: Status 404 returned error can't find the container with id 291477be58c963cb10395df09b5776b1569ed7b71ebcd2802947c00499969d35 Oct 31 00:54:56.787043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.786990 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" event={"ID":"e21029b5-0178-4c15-9013-61e8b08b7f90","Type":"ContainerStarted","Data":"e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00"} Oct 31 00:54:56.787546 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.787108 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:54:56.787546 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.787134 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" event={"ID":"e21029b5-0178-4c15-9013-61e8b08b7f90","Type":"ContainerStarted","Data":"291477be58c963cb10395df09b5776b1569ed7b71ebcd2802947c00499969d35"} Oct 31 00:54:56.791087 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.791053 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" event={"ID":"cbfe5036-6f6e-49e3-9332-d558dd54b375","Type":"ContainerStarted","Data":"77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf"} Oct 31 00:54:56.791241 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.791103 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" event={"ID":"cbfe5036-6f6e-49e3-9332-d558dd54b375","Type":"ContainerStarted","Data":"dea10d426c91772f09510726b67cd6f406e79e9466d22d241af49ea7647c27b0"} Oct 31 00:54:56.791241 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.791126 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:54:56.904135 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:56.904073 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" podStartSLOduration=1.904059691 podStartE2EDuration="1.904059691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:56.90016997 +0000 UTC m=+1589.102420403" watchObservedRunningTime="2026-10-31 00:54:56.904059691 +0000 UTC m=+1589.106310155" Oct 31 00:54:57.015728 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:54:57.015662 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" podStartSLOduration=2.015640297 podStartE2EDuration="2.015640297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:57.011133216 +0000 UTC m=+1589.213383650" watchObservedRunningTime="2026-10-31 00:54:57.015640297 +0000 UTC m=+1589.217890730" Oct 31 00:55:07.020722 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:55:07.020681 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:55:07.807773 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:07.807741 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:55:07.809213 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:07.809187 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:55:09.884806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:09.884059 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm"] Oct 31 00:55:09.884806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:09.884437 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" containerID="cri-o://92c28d01a9259b07002df5ea6ef5a7f492a5cd1e405511d49b706380bf56c2c9" gracePeriod=30 Oct 31 00:55:15.026407 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.026367 5890 generic.go:356] "Generic (PLEG): container finished" podID="682d7401-57c1-4cab-9893-8822207e3db6" containerID="92c28d01a9259b07002df5ea6ef5a7f492a5cd1e405511d49b706380bf56c2c9" exitCode=0 Oct 31 00:55:15.026765 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.026439 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" event={"ID":"682d7401-57c1-4cab-9893-8822207e3db6","Type":"ContainerDied","Data":"92c28d01a9259b07002df5ea6ef5a7f492a5cd1e405511d49b706380bf56c2c9"} Oct 31 00:55:15.177498 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.177474 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:55:15.217610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.217579 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-istio-token\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-istio-token\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.217827 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.217665 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-credential-socket\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.217827 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.217694 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-kube-api-access-nnq6g\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-kube-api-access-nnq6g\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.217985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.217885 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/682d7401-57c1-4cab-9893-8822207e3db6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d7401-57c1-4cab-9893-8822207e3db6-istiod-ca-cert\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.217985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.217905 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-envoy\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.217985 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.217933 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/682d7401-57c1-4cab-9893-8822207e3db6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d7401-57c1-4cab-9893-8822207e3db6-istio-podinfo\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.218138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.217999 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-credential-socket" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.218138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218049 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-data\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.218138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218097 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-socket\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.218299 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218170 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-certs\") pod \"682d7401-57c1-4cab-9893-8822207e3db6\" (UID: \"682d7401-57c1-4cab-9893-8822207e3db6\") " Oct 31 00:55:15.218299 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218257 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682d7401-57c1-4cab-9893-8822207e3db6-istiod-ca-cert" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:15.218406 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218341 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-data" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.218406 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218359 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-socket" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.218619 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218596 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-certs" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.218804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218766 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.218804 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218805 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.218957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218818 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.218957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218832 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.218957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.218844 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d7401-57c1-4cab-9893-8822207e3db6-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.220506 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.220482 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-kube-api-access-nnq6g" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "kube-api-access-nnq6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:15.220601 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.220518 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-istio-token" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:15.220738 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.220715 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-envoy" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.220865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.220737 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/682d7401-57c1-4cab-9893-8822207e3db6-istio-podinfo" pod "682d7401-57c1-4cab-9893-8822207e3db6" (UID: "682d7401-57c1-4cab-9893-8822207e3db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:15.320986 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.320941 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnq6g\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-kube-api-access-nnq6g\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.320986 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.320984 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d7401-57c1-4cab-9893-8822207e3db6-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.321243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.321000 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d7401-57c1-4cab-9893-8822207e3db6-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.321243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:15.321013 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d7401-57c1-4cab-9893-8822207e3db6-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:16.035969 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:16.035941 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" Oct 31 00:55:16.036452 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:16.035975 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm" event={"ID":"682d7401-57c1-4cab-9893-8822207e3db6","Type":"ContainerDied","Data":"4ecf510f1a18ce78c35887749cfacbcac220d76fce7e2630bf0df2567d89e8c6"} Oct 31 00:55:16.036452 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:16.036038 5890 scope.go:122] "RemoveContainer" containerID="92c28d01a9259b07002df5ea6ef5a7f492a5cd1e405511d49b706380bf56c2c9" Oct 31 00:55:16.247512 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:16.247415 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm"] Oct 31 00:55:16.294444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:16.294410 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-ljdnm"] Oct 31 00:55:17.936042 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:17.935989 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682d7401-57c1-4cab-9893-8822207e3db6" path="/var/lib/kubelet/pods/682d7401-57c1-4cab-9893-8822207e3db6/volumes" Oct 31 00:55:37.019244 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:55:37.019207 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:55:52.399727 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:52.399688 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j"] Oct 31 00:55:52.400200 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:52.399935 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" containerID="cri-o://4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4" gracePeriod=30 Oct 31 00:55:52.447190 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:52.447152 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9"] Oct 31 00:55:52.447378 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:52.447360 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" containerID="cri-o://5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b" gracePeriod=30 Oct 31 00:55:57.751426 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.751401 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:55:57.771083 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.771057 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:55:57.898168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898130 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0931745-893d-45b3-950d-1758093cc8ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0931745-893d-45b3-950d-1758093cc8ac-istio-podinfo\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898356 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898215 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-istio-token\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-istio-token\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898356 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898289 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-certs\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.898356 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898318 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-envoy\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898350 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-credential-socket\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898399 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istiod-ca-cert\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.898548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898437 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-data\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.898548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898463 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0931745-893d-45b3-950d-1758093cc8ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0931745-893d-45b3-950d-1758093cc8ac-istiod-ca-cert\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898548 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898496 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-podinfo\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.898829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898553 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-data\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898601 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-socket\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898649 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-certs\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.898829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898760 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-socket\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.899055 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898861 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-data" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.899055 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898869 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-kube-api-access-l9h84\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-kube-api-access-l9h84\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.899055 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898890 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-certs" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.899055 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.898972 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-token\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.899055 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899030 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-credential-socket\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.899324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899074 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-envoy\") pod \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\" (UID: \"385cf49c-32b3-4e38-acf7-b2c5ae63dc22\") " Oct 31 00:55:57.899324 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899119 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-kube-api-access-t27bn\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-kube-api-access-t27bn\") pod \"b0931745-893d-45b3-950d-1758093cc8ac\" (UID: \"b0931745-893d-45b3-950d-1758093cc8ac\") " Oct 31 00:55:57.899832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899115 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-credential-socket" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.899832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899154 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-data" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.899832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899211 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0931745-893d-45b3-950d-1758093cc8ac-istiod-ca-cert" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:57.899832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899339 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-certs" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.899832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899369 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-socket" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899460 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-socket" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899709 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-credential-socket" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.899997 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900018 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900034 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900048 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900063 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0931745-893d-45b3-950d-1758093cc8ac-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900083 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900102 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900115 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.900199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.900129 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.901883 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.901845 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-envoy" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.902195 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.902172 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-kube-api-access-t27bn" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "kube-api-access-t27bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.902596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.902573 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-token" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.902701 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.902686 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-podinfo" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:57.902870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.902847 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istiod-ca-cert" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:57.903225 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.903184 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-istio-token" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.903225 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.903189 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0931745-893d-45b3-950d-1758093cc8ac-istio-podinfo" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:57.903618 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.903599 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-kube-api-access-l9h84" pod "385cf49c-32b3-4e38-acf7-b2c5ae63dc22" (UID: "385cf49c-32b3-4e38-acf7-b2c5ae63dc22"). InnerVolumeSpecName "kube-api-access-l9h84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.904429 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:57.904401 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-envoy" pod "b0931745-893d-45b3-950d-1758093cc8ac" (UID: "b0931745-893d-45b3-950d-1758093cc8ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.001960 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.001991 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0931745-893d-45b3-950d-1758093cc8ac-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.002001 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.002010 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.002019 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9h84\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-kube-api-access-l9h84\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.002029 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.002039 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49c-32b3-4e38-acf7-b2c5ae63dc22-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.002048 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t27bn\" (UniqueName: \"kubernetes.io/projected/b0931745-893d-45b3-950d-1758093cc8ac-kube-api-access-t27bn\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.002407 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.002057 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0931745-893d-45b3-950d-1758093cc8ac-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.386623 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.386541 5890 generic.go:356] "Generic (PLEG): container finished" podID="b0931745-893d-45b3-950d-1758093cc8ac" containerID="4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4" exitCode=0 Oct 31 00:55:58.386623 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.386614 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" Oct 31 00:55:58.386843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.386611 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" event={"ID":"b0931745-893d-45b3-950d-1758093cc8ac","Type":"ContainerDied","Data":"4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4"} Oct 31 00:55:58.386843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.386778 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j" event={"ID":"b0931745-893d-45b3-950d-1758093cc8ac","Type":"ContainerDied","Data":"6285c272b318d38357df2cbfba6db0f48021c092ed7bf1678a9b86f48afcef0d"} Oct 31 00:55:58.386843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.386825 5890 scope.go:122] "RemoveContainer" containerID="4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4" Oct 31 00:55:58.388940 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.388915 5890 generic.go:356] "Generic (PLEG): container finished" podID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerID="5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b" exitCode=0 Oct 31 00:55:58.389338 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.388979 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" Oct 31 00:55:58.389338 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.388997 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" event={"ID":"385cf49c-32b3-4e38-acf7-b2c5ae63dc22","Type":"ContainerDied","Data":"5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b"} Oct 31 00:55:58.389338 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.389319 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9" event={"ID":"385cf49c-32b3-4e38-acf7-b2c5ae63dc22","Type":"ContainerDied","Data":"3be1e0db60177986b0602b91d2c97d9140a2fcc850a0c27dec002dd97138dcde"} Oct 31 00:55:58.399525 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.399506 5890 scope.go:122] "RemoveContainer" containerID="4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4" Oct 31 00:55:58.399802 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:55:58.399770 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4\": container with ID starting with 4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4 not found: ID does not exist" containerID="4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4" Oct 31 00:55:58.399874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.399809 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4"} err="failed to get container status \"4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4\": rpc error: code = NotFound desc = could not find container \"4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4\": container with ID starting with 4425e81dfb47a50b7506aec52a30794621ec96cdfded9695c29139b092e0d0a4 not found: ID does not exist" Oct 31 00:55:58.399874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.399826 5890 scope.go:122] "RemoveContainer" containerID="5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b" Oct 31 00:55:58.409550 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.409530 5890 scope.go:122] "RemoveContainer" containerID="5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b" Oct 31 00:55:58.409904 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:55:58.409884 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b\": container with ID starting with 5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b not found: ID does not exist" containerID="5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b" Oct 31 00:55:58.409962 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.409910 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b"} err="failed to get container status \"5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b\": rpc error: code = NotFound desc = could not find container \"5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b\": container with ID starting with 5fea2693bb9d09d55b3d9dd307f64b79d7a9c2c97fd1b9f859d284fbc506a65b not found: ID does not exist" Oct 31 00:55:58.546972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.546936 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j"] Oct 31 00:55:58.593938 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.593906 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-qwx5j"] Oct 31 00:55:58.658151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.658067 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9"] Oct 31 00:55:58.663610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:58.663580 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-gpwm9"] Oct 31 00:55:59.934019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:59.933990 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" path="/var/lib/kubelet/pods/385cf49c-32b3-4e38-acf7-b2c5ae63dc22/volumes" Oct 31 00:55:59.934483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:55:59.934467 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0931745-893d-45b3-950d-1758093cc8ac" path="/var/lib/kubelet/pods/b0931745-893d-45b3-950d-1758093cc8ac/volumes" Oct 31 00:56:00.053281 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.053244 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz"] Oct 31 00:56:00.053542 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.053513 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" containerID="cri-o://ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22" gracePeriod=30 Oct 31 00:56:00.101075 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.101039 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg"] Oct 31 00:56:00.101287 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.101265 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" containerID="cri-o://9ddd88f218e98276e2789922f04eacc4063e87be26c3969ce0f94b91893210b6" gracePeriod=30 Oct 31 00:56:00.106452 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.106424 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5"] Oct 31 00:56:00.106684 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.106655 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" containerID="cri-o://1f2a2b042d65fa1c2d9c1200e9367641b4f5db2b2fae4cd6ac94eec16fa42d56" gracePeriod=30 Oct 31 00:56:00.149687 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.149651 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv"] Oct 31 00:56:00.150190 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:00.150158 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" containerID="cri-o://03be9857d88c69a09cf0ef0ce37288a42580f954e445afd3a1f362fc07c91a9f" gracePeriod=30 Oct 31 00:56:05.474846 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.473447 5890 generic.go:356] "Generic (PLEG): container finished" podID="910c0d85-5973-47b2-84dd-08fb08117773" containerID="9ddd88f218e98276e2789922f04eacc4063e87be26c3969ce0f94b91893210b6" exitCode=0 Oct 31 00:56:05.474846 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.473524 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" event={"ID":"910c0d85-5973-47b2-84dd-08fb08117773","Type":"ContainerDied","Data":"9ddd88f218e98276e2789922f04eacc4063e87be26c3969ce0f94b91893210b6"} Oct 31 00:56:05.474846 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.474667 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:56:05.479835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.475542 5890 generic.go:356] "Generic (PLEG): container finished" podID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerID="1f2a2b042d65fa1c2d9c1200e9367641b4f5db2b2fae4cd6ac94eec16fa42d56" exitCode=0 Oct 31 00:56:05.479835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.475613 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" event={"ID":"f555bceb-b443-4267-b3e9-deaae7ad2e1a","Type":"ContainerDied","Data":"1f2a2b042d65fa1c2d9c1200e9367641b4f5db2b2fae4cd6ac94eec16fa42d56"} Oct 31 00:56:05.479835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.476901 5890 generic.go:356] "Generic (PLEG): container finished" podID="31426775-6b80-4ab7-9fb4-bffee896c601" containerID="ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22" exitCode=0 Oct 31 00:56:05.479835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.477038 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" event={"ID":"31426775-6b80-4ab7-9fb4-bffee896c601","Type":"ContainerDied","Data":"ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22"} Oct 31 00:56:05.479835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.477067 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" event={"ID":"31426775-6b80-4ab7-9fb4-bffee896c601","Type":"ContainerDied","Data":"770a703f92ebd2c43d14cf1736c3f26a808f456a9355ca8d76d4a91fe92490c3"} Oct 31 00:56:05.479835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.477089 5890 scope.go:122] "RemoveContainer" containerID="ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22" Oct 31 00:56:05.479835 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.477187 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz" Oct 31 00:56:05.481768 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.481741 5890 generic.go:356] "Generic (PLEG): container finished" podID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerID="03be9857d88c69a09cf0ef0ce37288a42580f954e445afd3a1f362fc07c91a9f" exitCode=0 Oct 31 00:56:05.481937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.481817 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" event={"ID":"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7","Type":"ContainerDied","Data":"03be9857d88c69a09cf0ef0ce37288a42580f954e445afd3a1f362fc07c91a9f"} Oct 31 00:56:05.518029 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.518000 5890 scope.go:122] "RemoveContainer" containerID="ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22" Oct 31 00:56:05.518322 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.518299 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-istio-token\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-istio-token\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.518402 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.518385 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-credential-socket\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.518552 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.518458 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-data\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.518762 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.518740 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-credential-socket" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.519146 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519110 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-data" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.519452 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:56:05.519420 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22\": container with ID starting with ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22 not found: ID does not exist" containerID="ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22" Oct 31 00:56:05.519557 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519461 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22"} err="failed to get container status \"ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22\": rpc error: code = NotFound desc = could not find container \"ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22\": container with ID starting with ec368caa0044df3ed68ed871360e89a9df0dd70e7f5e39f8f09f6234a35ebd22 not found: ID does not exist" Oct 31 00:56:05.519557 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519489 5890 scope.go:122] "RemoveContainer" containerID="0ebd8a6ccebe6eb6443fe36868eceec141883f3c653882c9aab600050d0d4015" Oct 31 00:56:05.519668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519569 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-certs\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.519668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519655 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31426775-6b80-4ab7-9fb4-bffee896c601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31426775-6b80-4ab7-9fb4-bffee896c601-istio-podinfo\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.519770 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519687 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-kube-api-access-55rx9\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-kube-api-access-55rx9\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.519890 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519778 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-socket\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.519948 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.519919 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31426775-6b80-4ab7-9fb4-bffee896c601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31426775-6b80-4ab7-9fb4-bffee896c601-istiod-ca-cert\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.520055 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.520041 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-envoy\") pod \"31426775-6b80-4ab7-9fb4-bffee896c601\" (UID: \"31426775-6b80-4ab7-9fb4-bffee896c601\") " Oct 31 00:56:05.520751 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.520691 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-certs" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.521263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.521139 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.521263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.521167 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.521263 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.521182 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.521531 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.521311 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-socket" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.521991 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.521952 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31426775-6b80-4ab7-9fb4-bffee896c601-istiod-ca-cert" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.522600 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.522576 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31426775-6b80-4ab7-9fb4-bffee896c601-istio-podinfo" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.523031 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.522999 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-istio-token" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.523115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.523040 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-kube-api-access-55rx9" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "kube-api-access-55rx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.524153 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.524129 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-envoy" pod "31426775-6b80-4ab7-9fb4-bffee896c601" (UID: "31426775-6b80-4ab7-9fb4-bffee896c601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.535104 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.535079 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:56:05.541615 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.541591 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:56:05.563874 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.563841 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:56:05.622175 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622093 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-data\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.622175 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622138 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-certs\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.622175 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622160 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-socket\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.622454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622207 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-envoy\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.622454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622229 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/910c0d85-5973-47b2-84dd-08fb08117773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/910c0d85-5973-47b2-84dd-08fb08117773-istio-podinfo\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.622454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622255 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-credential-socket\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.622454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622333 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-certs\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.622454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622389 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-data\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.622454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622426 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-credential-socket\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.622454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622434 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-data" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622465 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-credential-socket\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622439 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-certs" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622508 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-socket\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622526 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-socket" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622550 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-envoy\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622619 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istiod-ca-cert\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622633 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-certs" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622649 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-istio-token\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-istio-token\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622716 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-credential-socket" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622743 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-socket\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622770 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istiod-ca-cert\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622816 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-kube-api-access-nglvm\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-kube-api-access-nglvm\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.622832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622826 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-credential-socket" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622862 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-data" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622901 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-certs\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622957 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-token\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.622984 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-credential-socket" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623001 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-podinfo\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623030 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-envoy\") pod \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\" (UID: \"f555bceb-b443-4267-b3e9-deaae7ad2e1a\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623067 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-data\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623107 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/910c0d85-5973-47b2-84dd-08fb08117773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/910c0d85-5973-47b2-84dd-08fb08117773-istiod-ca-cert\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623155 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-kube-api-access-nw78f\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-kube-api-access-nw78f\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623173 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istiod-ca-cert" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623198 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-token\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-token\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623238 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-kube-api-access-l2z4v\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-kube-api-access-l2z4v\") pod \"910c0d85-5973-47b2-84dd-08fb08117773\" (UID: \"910c0d85-5973-47b2-84dd-08fb08117773\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623278 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-podinfo\") pod \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\" (UID: \"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7\") " Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623408 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-socket" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.623504 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623453 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-socket" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623644 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-data" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.623834 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-certs" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624046 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istiod-ca-cert" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624090 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624110 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624126 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624139 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624152 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624166 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624179 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624193 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624204 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624216 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624228 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31426775-6b80-4ab7-9fb4-bffee896c601-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624240 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624253 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55rx9\" (UniqueName: \"kubernetes.io/projected/31426775-6b80-4ab7-9fb4-bffee896c601-kube-api-access-55rx9\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624266 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624280 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624292 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.624297 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624305 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31426775-6b80-4ab7-9fb4-bffee896c601-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.625341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624317 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31426775-6b80-4ab7-9fb4-bffee896c601-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.625341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624331 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.625341 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.624390 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910c0d85-5973-47b2-84dd-08fb08117773-istiod-ca-cert" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.625570 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.625542 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-istio-token" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.626007 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.625941 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/910c0d85-5973-47b2-84dd-08fb08117773-istio-podinfo" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.626731 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.626706 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-envoy" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.627140 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.627104 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-token" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.627223 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.627134 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-kube-api-access-nglvm" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "kube-api-access-nglvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.627314 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.627292 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-token" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.627849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.627813 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-kube-api-access-nw78f" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "kube-api-access-nw78f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.627849 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.627840 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-envoy" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.628134 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.628116 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-envoy" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.628896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.628870 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-kube-api-access-l2z4v" pod "910c0d85-5973-47b2-84dd-08fb08117773" (UID: "910c0d85-5973-47b2-84dd-08fb08117773"). InnerVolumeSpecName "kube-api-access-l2z4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.629047 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.629025 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-podinfo" pod "f555bceb-b443-4267-b3e9-deaae7ad2e1a" (UID: "f555bceb-b443-4267-b3e9-deaae7ad2e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.629115 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.629043 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-podinfo" pod "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" (UID: "5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.725686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725647 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/910c0d85-5973-47b2-84dd-08fb08117773-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725680 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725690 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725700 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglvm\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-kube-api-access-nglvm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725712 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725720 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725729 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f555bceb-b443-4267-b3e9-deaae7ad2e1a-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725738 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/910c0d85-5973-47b2-84dd-08fb08117773-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725747 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw78f\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-kube-api-access-nw78f\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725756 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725764 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2z4v\" (UniqueName: \"kubernetes.io/projected/910c0d85-5973-47b2-84dd-08fb08117773-kube-api-access-l2z4v\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725772 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725796 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.725934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:05.725808 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/910c0d85-5973-47b2-84dd-08fb08117773-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:06.168843 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.168755 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz"] Oct 31 00:56:06.173196 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.173165 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-r8sjz"] Oct 31 00:56:06.491705 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.491673 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" Oct 31 00:56:06.492265 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.491687 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg" event={"ID":"910c0d85-5973-47b2-84dd-08fb08117773","Type":"ContainerDied","Data":"6149ed5b54646b8f4db104de5c443f989443be481e60fcbaa471868b11b07555"} Oct 31 00:56:06.492429 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.492414 5890 scope.go:122] "RemoveContainer" containerID="9ddd88f218e98276e2789922f04eacc4063e87be26c3969ce0f94b91893210b6" Oct 31 00:56:06.494025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.493859 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" Oct 31 00:56:06.494025 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.493903 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5" event={"ID":"f555bceb-b443-4267-b3e9-deaae7ad2e1a","Type":"ContainerDied","Data":"8d216c80be7118387f707c3e42b0f7e2ff8c8a48eb9d5d4dbf1938bfad77b703"} Oct 31 00:56:06.497583 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.497559 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" event={"ID":"5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7","Type":"ContainerDied","Data":"2e3d2431df6d326483bff329527280f6bfae038c32d0030254d6641bca145fad"} Oct 31 00:56:06.497721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.497609 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv" Oct 31 00:56:06.503741 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.503723 5890 scope.go:122] "RemoveContainer" containerID="1f2a2b042d65fa1c2d9c1200e9367641b4f5db2b2fae4cd6ac94eec16fa42d56" Oct 31 00:56:06.513020 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.513002 5890 scope.go:122] "RemoveContainer" containerID="03be9857d88c69a09cf0ef0ce37288a42580f954e445afd3a1f362fc07c91a9f" Oct 31 00:56:06.651415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.651378 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5"] Oct 31 00:56:06.658529 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.658496 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-64pl5"] Oct 31 00:56:06.763152 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.763053 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg"] Oct 31 00:56:06.766864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.766836 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-ckvvg"] Oct 31 00:56:06.938330 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.938302 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv"] Oct 31 00:56:06.941832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:06.941809 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-8lrwv"] Oct 31 00:56:07.019325 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:56:07.019238 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:56:07.934914 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:07.934884 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" path="/var/lib/kubelet/pods/31426775-6b80-4ab7-9fb4-bffee896c601/volumes" Oct 31 00:56:07.935405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:07.935390 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" path="/var/lib/kubelet/pods/5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7/volumes" Oct 31 00:56:07.935850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:07.935837 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="910c0d85-5973-47b2-84dd-08fb08117773" path="/var/lib/kubelet/pods/910c0d85-5973-47b2-84dd-08fb08117773/volumes" Oct 31 00:56:07.936569 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:07.936554 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" path="/var/lib/kubelet/pods/f555bceb-b443-4267-b3e9-deaae7ad2e1a/volumes" Oct 31 00:56:27.687387 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:27.687348 5890 generic.go:356] "Generic (PLEG): container finished" podID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerID="52cc85aae700c11dc46b80e04f7ce29e6a96151b99525447bf4a1bcf655c59a7" exitCode=0 Oct 31 00:56:27.687853 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:27.687402 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" event={"ID":"b64b373e-2051-4517-b1c0-a25c5f61d217","Type":"ContainerDied","Data":"52cc85aae700c11dc46b80e04f7ce29e6a96151b99525447bf4a1bcf655c59a7"} Oct 31 00:56:27.687853 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:27.687833 5890 scope.go:122] "RemoveContainer" containerID="52cc85aae700c11dc46b80e04f7ce29e6a96151b99525447bf4a1bcf655c59a7" Oct 31 00:56:27.689684 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:27.689643 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:27.689797 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:27.689766 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:27.689850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:27.689834 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:28.823836 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:28.823273 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" event={"ID":"b64b373e-2051-4517-b1c0-a25c5f61d217","Type":"ContainerStarted","Data":"e228b0e08592371c30cc27e328bccdd9ffbadd9744ed6badb9ecf66d26e4b171"} Oct 31 00:56:29.597927 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:29.597885 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:56:29.599498 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:29.599467 5890 patch_prober.go:29] interesting pod/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:56:29.599629 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:29.599523 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 00:56:30.598311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:30.598273 5890 patch_prober.go:29] interesting pod/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:56:30.598737 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:30.598333 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 00:56:31.598698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:31.598660 5890 patch_prober.go:29] interesting pod/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:56:31.599142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:31.598729 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 00:56:32.599014 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:32.598972 5890 patch_prober.go:29] interesting pod/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:56:32.599657 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:32.599045 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 00:56:33.341815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:33.340772 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:56:33.598564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:33.598484 5890 patch_prober.go:29] interesting pod/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:56:33.598564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:33.598541 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 00:56:34.602564 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.602530 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:56:34.603704 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.603680 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 00:56:34.910770 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.910685 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7"] Oct 31 00:56:34.911650 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911626 5890 cpu_manager.go:414] "containerMap: removing container" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" Oct 31 00:56:34.911650 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911651 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" Oct 31 00:56:34.911864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911668 5890 cpu_manager.go:414] "containerMap: removing container" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" Oct 31 00:56:34.911864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911677 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" Oct 31 00:56:34.911864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911702 5890 cpu_manager.go:414] "containerMap: removing container" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" Oct 31 00:56:34.911864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911710 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" Oct 31 00:56:34.911864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911743 5890 cpu_manager.go:414] "containerMap: removing container" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" Oct 31 00:56:34.911864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911751 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" Oct 31 00:56:34.911864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911775 5890 cpu_manager.go:414] "containerMap: removing container" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911885 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911985 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.911995 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912041 5890 cpu_manager.go:414] "containerMap: removing container" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912051 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912246 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912257 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="910c0d85-5973-47b2-84dd-08fb08117773" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912272 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="f555bceb-b443-4267-b3e9-deaae7ad2e1a" containerName="istio-proxy" Oct 31 00:56:34.912288 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912285 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="31426775-6b80-4ab7-9fb4-bffee896c601" containerName="istio-proxy" Oct 31 00:56:34.912808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912298 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="682d7401-57c1-4cab-9893-8822207e3db6" containerName="istio-proxy" Oct 31 00:56:34.912808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912313 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0931745-893d-45b3-950d-1758093cc8ac" containerName="istio-proxy" Oct 31 00:56:34.912808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912325 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" Oct 31 00:56:34.912808 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.912334 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="385cf49c-32b3-4e38-acf7-b2c5ae63dc22" containerName="istio-proxy" Oct 31 00:56:34.918028 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.918008 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:34.968438 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.968403 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7e6fe88b-openshift-default-dockercfg-c2g5f\"" Oct 31 00:56:34.999005 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:34.998971 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7"] Oct 31 00:56:35.072159 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072118 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e88b93-8cef-471f-858b-45b842de5656-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.072329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072165 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.072329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072244 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e88b93-8cef-471f-858b-45b842de5656-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.072428 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072414 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.072554 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072526 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.072629 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072610 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.072898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072872 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.073022 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072933 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmdm\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-kube-api-access-wgmdm\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.073022 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.072987 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.075287 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.075264 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf"] Oct 31 00:56:35.075896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.075881 5890 cpu_manager.go:414] "containerMap: removing container" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" Oct 31 00:56:35.075964 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.075898 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6e2532-2e9b-468b-8c2b-9a0fbafe66c7" containerName="istio-proxy" Oct 31 00:56:35.079600 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.079582 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.130926 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.130687 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cb2b8a\"" Oct 31 00:56:35.134893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.134859 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf"] Oct 31 00:56:35.174192 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174163 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174222 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174297 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce1d06-269d-4bdb-90bc-0e07f11a164e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.174344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174332 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174369 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmdm\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-kube-api-access-wgmdm\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174390 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174497 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e88b93-8cef-471f-858b-45b842de5656-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174667 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174533 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174667 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174561 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e88b93-8cef-471f-858b-45b842de5656-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174667 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174659 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.174893 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174746 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85ws\" (UniqueName: \"kubernetes.io/projected/6bce1d06-269d-4bdb-90bc-0e07f11a164e-kube-api-access-k85ws\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.174981 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.174945 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.175086 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.175057 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.175241 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.175220 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.175315 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.175277 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e88b93-8cef-471f-858b-45b842de5656-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.175315 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.175286 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.176948 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.176930 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.177289 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.177271 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e88b93-8cef-471f-858b-45b842de5656-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.241805 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.241122 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmdm\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-kube-api-access-wgmdm\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.244879 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.244427 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.244879 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.244672 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:35.288726 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.285813 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85ws\" (UniqueName: \"kubernetes.io/projected/6bce1d06-269d-4bdb-90bc-0e07f11a164e-kube-api-access-k85ws\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.288726 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.285981 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce1d06-269d-4bdb-90bc-0e07f11a164e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.288726 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.287141 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce1d06-269d-4bdb-90bc-0e07f11a164e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.348884 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.348848 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85ws\" (UniqueName: \"kubernetes.io/projected/6bce1d06-269d-4bdb-90bc-0e07f11a164e-kube-api-access-k85ws\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.393926 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.393699 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:35.742016 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.741991 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7"] Oct 31 00:56:35.742982 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:56:35.742952 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e88b93_8cef_471f_858b_45b842de5656.slice/crio-d8258e686126c8445c57dcd896e059af0aa0fdc89a94d3cc288817340a1e3b6b WatchSource:0}: Error finding container d8258e686126c8445c57dcd896e059af0aa0fdc89a94d3cc288817340a1e3b6b: Status 404 returned error can't find the container with id d8258e686126c8445c57dcd896e059af0aa0fdc89a94d3cc288817340a1e3b6b Oct 31 00:56:35.745553 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.745516 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:35.745648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.745597 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:35.745719 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.745643 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:35.787889 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.787865 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf"] Oct 31 00:56:35.790416 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:56:35.790386 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bce1d06_269d_4bdb_90bc_0e07f11a164e.slice/crio-10bf95891264a30a90b3ebf3de7859423fc4be2245705d7d248eaac1f39336ad WatchSource:0}: Error finding container 10bf95891264a30a90b3ebf3de7859423fc4be2245705d7d248eaac1f39336ad: Status 404 returned error can't find the container with id 10bf95891264a30a90b3ebf3de7859423fc4be2245705d7d248eaac1f39336ad Oct 31 00:56:35.918740 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.918692 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" event={"ID":"49e88b93-8cef-471f-858b-45b842de5656","Type":"ContainerStarted","Data":"ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c"} Oct 31 00:56:35.918981 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.918760 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" event={"ID":"49e88b93-8cef-471f-858b-45b842de5656","Type":"ContainerStarted","Data":"d8258e686126c8445c57dcd896e059af0aa0fdc89a94d3cc288817340a1e3b6b"} Oct 31 00:56:35.920547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.920505 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" event={"ID":"6bce1d06-269d-4bdb-90bc-0e07f11a164e","Type":"ContainerStarted","Data":"ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8"} Oct 31 00:56:35.920652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.920563 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" event={"ID":"6bce1d06-269d-4bdb-90bc-0e07f11a164e","Type":"ContainerStarted","Data":"10bf95891264a30a90b3ebf3de7859423fc4be2245705d7d248eaac1f39336ad"} Oct 31 00:56:35.920652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:35.920595 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:36.035158 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:36.035056 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podStartSLOduration=2.035040843 podStartE2EDuration="2.035040843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:36.034345539 +0000 UTC m=+1688.236595977" watchObservedRunningTime="2026-10-31 00:56:36.035040843 +0000 UTC m=+1688.237291270" Oct 31 00:56:36.150541 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:36.150486 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" podStartSLOduration=1.150472252 podStartE2EDuration="1.150472252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:36.145722814 +0000 UTC m=+1688.347973245" watchObservedRunningTime="2026-10-31 00:56:36.150472252 +0000 UTC m=+1688.352722723" Oct 31 00:56:36.250133 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:36.250100 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:36.255394 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:36.255351 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:36.255594 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:36.255428 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:56:37.020808 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:56:37.020549 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:56:37.250814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:37.250365 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:37.250814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:37.250428 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:56:38.268811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:38.265176 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:38.268811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:38.265241 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:56:39.258810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:39.253884 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:39.258810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:39.253949 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:56:40.247811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:40.246109 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:40.247811 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:40.246171 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:56:41.261192 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:41.261152 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:41.263081 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:41.261756 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 00:56:42.270834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:42.268464 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:42.270834 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:42.269399 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:42.271572 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:42.271545 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:56:42.973622 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:42.973589 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r"] Oct 31 00:56:42.973861 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:42.973828 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" podUID="70f6493a-ab3c-42bc-845d-daed532cb047" containerName="maas-api" containerID="cri-o://f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163" gracePeriod=30 Oct 31 00:56:43.288188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.288154 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:56:43.333521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.333491 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls\") pod \"70f6493a-ab3c-42bc-845d-daed532cb047\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " Oct 31 00:56:43.333660 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.333562 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-metrics-certs\") pod \"70f6493a-ab3c-42bc-845d-daed532cb047\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " Oct 31 00:56:43.333660 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.333628 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f6493a-ab3c-42bc-845d-daed532cb047-kube-api-access-4jj29\" (UniqueName: \"kubernetes.io/projected/70f6493a-ab3c-42bc-845d-daed532cb047-kube-api-access-4jj29\") pod \"70f6493a-ab3c-42bc-845d-daed532cb047\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " Oct 31 00:56:43.333823 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.333777 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f6493a-ab3c-42bc-845d-daed532cb047-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f6493a-ab3c-42bc-845d-daed532cb047-openshift-service-ca\") pod \"70f6493a-ab3c-42bc-845d-daed532cb047\" (UID: \"70f6493a-ab3c-42bc-845d-daed532cb047\") " Oct 31 00:56:43.334138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.334110 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f6493a-ab3c-42bc-845d-daed532cb047-openshift-service-ca" pod "70f6493a-ab3c-42bc-845d-daed532cb047" (UID: "70f6493a-ab3c-42bc-845d-daed532cb047"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:43.334466 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.334447 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f6493a-ab3c-42bc-845d-daed532cb047-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.335653 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.335621 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls" pod "70f6493a-ab3c-42bc-845d-daed532cb047" (UID: "70f6493a-ab3c-42bc-845d-daed532cb047"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:43.335748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.335695 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-metrics-certs" pod "70f6493a-ab3c-42bc-845d-daed532cb047" (UID: "70f6493a-ab3c-42bc-845d-daed532cb047"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:43.335822 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.335776 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f6493a-ab3c-42bc-845d-daed532cb047-kube-api-access-4jj29" pod "70f6493a-ab3c-42bc-845d-daed532cb047" (UID: "70f6493a-ab3c-42bc-845d-daed532cb047"). InnerVolumeSpecName "kube-api-access-4jj29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:43.435887 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.435804 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.435887 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.435842 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f6493a-ab3c-42bc-845d-daed532cb047-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.435887 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:43.435858 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jj29\" (UniqueName: \"kubernetes.io/projected/70f6493a-ab3c-42bc-845d-daed532cb047-kube-api-access-4jj29\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:44.091527 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.082759 5890 generic.go:356] "Generic (PLEG): container finished" podID="70f6493a-ab3c-42bc-845d-daed532cb047" containerID="f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163" exitCode=0 Oct 31 00:56:44.099240 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.085889 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" event={"ID":"70f6493a-ab3c-42bc-845d-daed532cb047","Type":"ContainerDied","Data":"f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163"} Oct 31 00:56:44.099485 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.099462 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" event={"ID":"70f6493a-ab3c-42bc-845d-daed532cb047","Type":"ContainerDied","Data":"3ef822e70c89dd150859542668d825057ce94c6b51d60adbd24ae1bcd90a4cf0"} Oct 31 00:56:44.099614 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.099600 5890 scope.go:122] "RemoveContainer" containerID="f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163" Oct 31 00:56:44.099828 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.085976 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r" Oct 31 00:56:44.130032 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.129966 5890 scope.go:122] "RemoveContainer" containerID="f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163" Oct 31 00:56:44.130507 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:56:44.130484 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163\": container with ID starting with f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163 not found: ID does not exist" containerID="f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163" Oct 31 00:56:44.130640 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.130618 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163"} err="failed to get container status \"f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163\": rpc error: code = NotFound desc = could not find container \"f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163\": container with ID starting with f36ee1478bd123180d4c07bb770b06c2cb702adb9374eb2aace2ebfc671fe163 not found: ID does not exist" Oct 31 00:56:44.305057 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.304961 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r"] Oct 31 00:56:44.308684 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:44.308659 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69a4bc-54d96f98db-dwj9r"] Oct 31 00:56:45.936232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:45.936192 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f6493a-ab3c-42bc-845d-daed532cb047" path="/var/lib/kubelet/pods/70f6493a-ab3c-42bc-845d-daed532cb047/volumes" Oct 31 00:56:46.934837 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:46.934808 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:56:50.022870 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.022837 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx"] Oct 31 00:56:50.023475 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.023459 5890 cpu_manager.go:414] "containerMap: removing container" podUID="70f6493a-ab3c-42bc-845d-daed532cb047" containerName="maas-api" Oct 31 00:56:50.023520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.023476 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f6493a-ab3c-42bc-845d-daed532cb047" containerName="maas-api" Oct 31 00:56:50.023618 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.023609 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f6493a-ab3c-42bc-845d-daed532cb047" containerName="maas-api" Oct 31 00:56:50.027149 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.027131 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.070304 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.070274 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss"] Oct 31 00:56:50.074406 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.074387 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.125181 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.125149 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx"] Oct 31 00:56:50.135721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.135682 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.135899 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.135742 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.135899 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.135776 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.135899 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.135815 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.135899 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.135861 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.135899 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.135889 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcl77\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-kube-api-access-tcl77\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136180 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.135961 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7xt\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-kube-api-access-7n7xt\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.136180 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136003 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136180 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136135 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.136180 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136175 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136213 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136239 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136263 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136295 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.136435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136323 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136371 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c97d1e-e95f-4828-b25c-181d42d3ea97-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.136435 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136414 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.136810 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.136455 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.180569 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.180533 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss"] Oct 31 00:56:50.237236 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237201 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.237433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237249 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.237433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237282 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.237433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237307 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.237433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237326 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.237433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237357 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.237433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237392 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237448 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c97d1e-e95f-4828-b25c-181d42d3ea97-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237488 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237526 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237592 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237629 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237667 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237692 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237698 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237732 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237754 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237805 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcl77\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-kube-api-access-tcl77\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237877 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7xt\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-kube-api-access-7n7xt\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.237923 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.238306 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.238094 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.239060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.238354 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.239060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.238516 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c97d1e-e95f-4828-b25c-181d42d3ea97-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.239060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.238827 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.239060 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.239025 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.239289 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.239118 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.239346 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.239296 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.239693 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.239662 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.240376 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.240354 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.240762 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.240744 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.241168 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.241148 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.241295 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.241276 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.298686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.298601 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.299170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.299146 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.299807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.299766 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7xt\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-kube-api-access-7n7xt\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.302188 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.302093 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcl77\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-kube-api-access-tcl77\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.342062 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.342033 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:50.389364 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.389329 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:50.777764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.777731 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx"] Oct 31 00:56:50.778417 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:56:50.778394 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c97d1e_e95f_4828_b25c_181d42d3ea97.slice/crio-6c637aa3de3adcf3dd0a22d94757c9ce4d1b4a0f50ea8d1791d19e682a993985 WatchSource:0}: Error finding container 6c637aa3de3adcf3dd0a22d94757c9ce4d1b4a0f50ea8d1791d19e682a993985: Status 404 returned error can't find the container with id 6c637aa3de3adcf3dd0a22d94757c9ce4d1b4a0f50ea8d1791d19e682a993985 Oct 31 00:56:50.781259 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.781217 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:50.781373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.781294 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:50.781373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.781339 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:50.823165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.823139 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss"] Oct 31 00:56:50.825028 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:56:50.824998 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8e3c09_4b4d_499a_b38a_c5a18870579a.slice/crio-7bb2fdef5f9d5d5285ec50c0b895e708d63b52b1235196a11b5dbff4b7d63bff WatchSource:0}: Error finding container 7bb2fdef5f9d5d5285ec50c0b895e708d63b52b1235196a11b5dbff4b7d63bff: Status 404 returned error can't find the container with id 7bb2fdef5f9d5d5285ec50c0b895e708d63b52b1235196a11b5dbff4b7d63bff Oct 31 00:56:50.827666 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.827635 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:50.827745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.827705 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:50.827745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:50.827739 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:51.161228 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.161118 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" event={"ID":"6e8e3c09-4b4d-499a-b38a-c5a18870579a","Type":"ContainerStarted","Data":"6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34"} Oct 31 00:56:51.161749 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.161720 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" event={"ID":"6e8e3c09-4b4d-499a-b38a-c5a18870579a","Type":"ContainerStarted","Data":"7bb2fdef5f9d5d5285ec50c0b895e708d63b52b1235196a11b5dbff4b7d63bff"} Oct 31 00:56:51.163973 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.163940 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" event={"ID":"88c97d1e-e95f-4828-b25c-181d42d3ea97","Type":"ContainerStarted","Data":"b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd"} Oct 31 00:56:51.164091 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.163997 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" event={"ID":"88c97d1e-e95f-4828-b25c-181d42d3ea97","Type":"ContainerStarted","Data":"6c637aa3de3adcf3dd0a22d94757c9ce4d1b4a0f50ea8d1791d19e682a993985"} Oct 31 00:56:51.283102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.283034 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podStartSLOduration=1.283013375 podStartE2EDuration="1.283013375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:51.280310709 +0000 UTC m=+1703.482561142" watchObservedRunningTime="2026-10-31 00:56:51.283013375 +0000 UTC m=+1703.485263810" Oct 31 00:56:51.342397 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.342359 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:51.343934 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.343904 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:56:51.344074 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.343963 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 00:56:51.397806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.391682 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:51.397806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.391871 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:56:51.397806 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.391916 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:56:51.413116 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:51.413006 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podStartSLOduration=2.412989255 podStartE2EDuration="2.412989255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:51.402004886 +0000 UTC m=+1703.604255340" watchObservedRunningTime="2026-10-31 00:56:51.412989255 +0000 UTC m=+1703.615239756" Oct 31 00:56:52.343138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:52.343094 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:56:52.343521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:52.343150 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 00:56:52.393831 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:52.393199 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:56:52.393831 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:52.393263 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:56:53.343019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:53.342547 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:56:53.343019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:53.342607 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 00:56:53.406659 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:53.406536 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:56:53.406659 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:53.406601 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:56:53.928146 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:53.928111 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2"] Oct 31 00:56:54.023499 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:54.023465 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt"] Oct 31 00:56:54.436802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:54.435019 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:56:54.436802 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:54.435081 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:56:54.442480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:54.442454 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:56:54.442669 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:54.442650 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 00:56:54.933705 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:54.933664 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c"] Oct 31 00:56:54.934068 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:54.934034 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" containerID="cri-o://77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715" gracePeriod=30 Oct 31 00:56:55.358619 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:55.358517 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:56:55.358924 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:55.358876 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 00:56:55.394262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:55.394146 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:56:55.394262 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:55.394204 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:56:56.356237 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:56.356193 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:56:56.360905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:56.360865 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 00:56:56.411913 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:56.411870 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:56:56.412197 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:56.412167 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:56:57.374575 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:57.374537 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:56:57.375600 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:57.375278 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 00:56:57.390229 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:57.390190 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:56:57.390545 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:57.390484 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 00:56:58.561775 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:58.561636 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:58.562403 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:58.562379 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:58.563496 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:58.563476 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:56:58.566422 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:58.566397 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:58.566904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:58.566885 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:58.567642 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:58.567620 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:56:58.936139 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:58.936100 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" podUID="e21029b5-0178-4c15-9013-61e8b08b7f90" containerName="payload-processing" containerID="cri-o://e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00" gracePeriod=25 Oct 31 00:56:59.024482 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.024439 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" podUID="cbfe5036-6f6e-49e3-9332-d558dd54b375" containerName="payload-pre-processing" containerID="cri-o://77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf" gracePeriod=25 Oct 31 00:56:59.350709 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.350681 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:56:59.379918 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.379894 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:56:59.390171 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.390143 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21029b5-0178-4c15-9013-61e8b08b7f90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21029b5-0178-4c15-9013-61e8b08b7f90-plugins-config-volume\") pod \"e21029b5-0178-4c15-9013-61e8b08b7f90\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " Oct 31 00:56:59.390304 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.390278 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21029b5-0178-4c15-9013-61e8b08b7f90-kube-api-access-9t8x8\" (UniqueName: \"kubernetes.io/projected/e21029b5-0178-4c15-9013-61e8b08b7f90-kube-api-access-9t8x8\") pod \"e21029b5-0178-4c15-9013-61e8b08b7f90\" (UID: \"e21029b5-0178-4c15-9013-61e8b08b7f90\") " Oct 31 00:56:59.390537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.390511 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21029b5-0178-4c15-9013-61e8b08b7f90-plugins-config-volume" pod "e21029b5-0178-4c15-9013-61e8b08b7f90" (UID: "e21029b5-0178-4c15-9013-61e8b08b7f90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:59.390696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.390678 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21029b5-0178-4c15-9013-61e8b08b7f90-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:59.392280 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.392262 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21029b5-0178-4c15-9013-61e8b08b7f90-kube-api-access-9t8x8" pod "e21029b5-0178-4c15-9013-61e8b08b7f90" (UID: "e21029b5-0178-4c15-9013-61e8b08b7f90"). InnerVolumeSpecName "kube-api-access-9t8x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:59.444320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.444290 5890 generic.go:356] "Generic (PLEG): container finished" podID="e21029b5-0178-4c15-9013-61e8b08b7f90" containerID="e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00" exitCode=0 Oct 31 00:56:59.444476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.444334 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" Oct 31 00:56:59.444476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.444373 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" event={"ID":"e21029b5-0178-4c15-9013-61e8b08b7f90","Type":"ContainerDied","Data":"e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00"} Oct 31 00:56:59.444476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.444429 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2" event={"ID":"e21029b5-0178-4c15-9013-61e8b08b7f90","Type":"ContainerDied","Data":"291477be58c963cb10395df09b5776b1569ed7b71ebcd2802947c00499969d35"} Oct 31 00:56:59.444476 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.444454 5890 scope.go:122] "RemoveContainer" containerID="e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00" Oct 31 00:56:59.445896 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.445868 5890 generic.go:356] "Generic (PLEG): container finished" podID="cbfe5036-6f6e-49e3-9332-d558dd54b375" containerID="77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf" exitCode=0 Oct 31 00:56:59.446035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.445913 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" Oct 31 00:56:59.446035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.445951 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" event={"ID":"cbfe5036-6f6e-49e3-9332-d558dd54b375","Type":"ContainerDied","Data":"77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf"} Oct 31 00:56:59.446035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.445996 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt" event={"ID":"cbfe5036-6f6e-49e3-9332-d558dd54b375","Type":"ContainerDied","Data":"dea10d426c91772f09510726b67cd6f406e79e9466d22d241af49ea7647c27b0"} Oct 31 00:56:59.455065 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.455046 5890 scope.go:122] "RemoveContainer" containerID="e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00" Oct 31 00:56:59.455321 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:56:59.455299 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00\": container with ID starting with e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00 not found: ID does not exist" containerID="e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00" Oct 31 00:56:59.455398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.455332 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00"} err="failed to get container status \"e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00\": rpc error: code = NotFound desc = could not find container \"e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00\": container with ID starting with e00d06d49f8adaeac17ba3a7a7ecf5dbab8cfa3d41d1e325872aebbf813cac00 not found: ID does not exist" Oct 31 00:56:59.455398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.455355 5890 scope.go:122] "RemoveContainer" containerID="77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf" Oct 31 00:56:59.464107 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.464086 5890 scope.go:122] "RemoveContainer" containerID="77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf" Oct 31 00:56:59.464340 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:56:59.464322 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf\": container with ID starting with 77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf not found: ID does not exist" containerID="77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf" Oct 31 00:56:59.464382 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.464346 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf"} err="failed to get container status \"77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf\": rpc error: code = NotFound desc = could not find container \"77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf\": container with ID starting with 77feecab92283389ccbaf3f905df1525b770e33f2ada4d87982edd344bca06cf not found: ID does not exist" Oct 31 00:56:59.492172 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.492100 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbfe5036-6f6e-49e3-9332-d558dd54b375-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfe5036-6f6e-49e3-9332-d558dd54b375-plugins-config-volume\") pod \"cbfe5036-6f6e-49e3-9332-d558dd54b375\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " Oct 31 00:56:59.492304 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.492202 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfe5036-6f6e-49e3-9332-d558dd54b375-kube-api-access-zrclt\" (UniqueName: \"kubernetes.io/projected/cbfe5036-6f6e-49e3-9332-d558dd54b375-kube-api-access-zrclt\") pod \"cbfe5036-6f6e-49e3-9332-d558dd54b375\" (UID: \"cbfe5036-6f6e-49e3-9332-d558dd54b375\") " Oct 31 00:56:59.492520 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.492498 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfe5036-6f6e-49e3-9332-d558dd54b375-plugins-config-volume" pod "cbfe5036-6f6e-49e3-9332-d558dd54b375" (UID: "cbfe5036-6f6e-49e3-9332-d558dd54b375"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:59.493162 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.493036 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfe5036-6f6e-49e3-9332-d558dd54b375-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:59.493162 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.493060 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8x8\" (UniqueName: \"kubernetes.io/projected/e21029b5-0178-4c15-9013-61e8b08b7f90-kube-api-access-9t8x8\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:59.494539 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.494513 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfe5036-6f6e-49e3-9332-d558dd54b375-kube-api-access-zrclt" pod "cbfe5036-6f6e-49e3-9332-d558dd54b375" (UID: "cbfe5036-6f6e-49e3-9332-d558dd54b375"). InnerVolumeSpecName "kube-api-access-zrclt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:59.581439 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.581363 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2"] Oct 31 00:56:59.586099 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.586075 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e6fe88b-558b6b9b97-6zdb2"] Oct 31 00:56:59.594420 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:56:59.594391 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrclt\" (UniqueName: \"kubernetes.io/projected/cbfe5036-6f6e-49e3-9332-d558dd54b375-kube-api-access-zrclt\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.003243 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.003207 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21029b5-0178-4c15-9013-61e8b08b7f90" path="/var/lib/kubelet/pods/e21029b5-0178-4c15-9013-61e8b08b7f90/volumes" Oct 31 00:57:00.003692 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.003669 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt"] Oct 31 00:57:00.003829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.003712 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e6fe88b-6c89966b75-x5kqt"] Oct 31 00:57:00.223794 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.223754 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:57:00.352454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352354 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-credential-socket\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352411 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-certs\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352454 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352442 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-data\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352545 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-envoy\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352575 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-socket\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352613 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-token\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-token\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352654 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-credential-socket" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.352752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352688 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istiod-ca-cert\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352728 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-podinfo\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.352752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352750 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-data" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.353120 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352767 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-kube-api-access-2rwdf\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-kube-api-access-2rwdf\") pod \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\" (UID: \"de0b0377-cdf7-4f36-9af5-d5772a1dba9c\") " Oct 31 00:57:00.353120 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352769 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-certs" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.353120 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.352881 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-socket" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.353120 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.353047 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istiod-ca-cert" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:00.353434 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.353410 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.353512 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.353438 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.353512 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.353453 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.353512 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.353468 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.353512 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.353480 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.355977 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.355953 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-token" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:00.356741 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.356717 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-envoy" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.356859 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.356744 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-podinfo" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:57:00.357029 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.357006 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-kube-api-access-2rwdf" pod "de0b0377-cdf7-4f36-9af5-d5772a1dba9c" (UID: "de0b0377-cdf7-4f36-9af5-d5772a1dba9c"). InnerVolumeSpecName "kube-api-access-2rwdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:00.455195 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.455159 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.455195 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.455194 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.455415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.455212 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.455415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.455228 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rwdf\" (UniqueName: \"kubernetes.io/projected/de0b0377-cdf7-4f36-9af5-d5772a1dba9c-kube-api-access-2rwdf\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.458056 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.458026 5890 generic.go:356] "Generic (PLEG): container finished" podID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerID="77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715" exitCode=0 Oct 31 00:57:00.458221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.458081 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" Oct 31 00:57:00.458221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.458116 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" event={"ID":"de0b0377-cdf7-4f36-9af5-d5772a1dba9c","Type":"ContainerDied","Data":"77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715"} Oct 31 00:57:00.458221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.458171 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c" event={"ID":"de0b0377-cdf7-4f36-9af5-d5772a1dba9c","Type":"ContainerDied","Data":"67491d3fada784830500d05d9c61f3263b304212b582620875ff83e789e9b9db"} Oct 31 00:57:00.458221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.458202 5890 scope.go:122] "RemoveContainer" containerID="77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715" Oct 31 00:57:00.471539 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.471438 5890 scope.go:122] "RemoveContainer" containerID="bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264" Oct 31 00:57:00.482389 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.482362 5890 scope.go:122] "RemoveContainer" containerID="77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715" Oct 31 00:57:00.482703 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:57:00.482673 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715\": container with ID starting with 77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715 not found: ID does not exist" containerID="77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715" Oct 31 00:57:00.482799 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.482711 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715"} err="failed to get container status \"77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715\": rpc error: code = NotFound desc = could not find container \"77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715\": container with ID starting with 77a4ec4b18b9e710e58a3cd313cd9d0502e9fcf708bc92710a80f44fb9c35715 not found: ID does not exist" Oct 31 00:57:00.482799 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.482729 5890 scope.go:122] "RemoveContainer" containerID="bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264" Oct 31 00:57:00.483100 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:57:00.483070 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264\": container with ID starting with bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264 not found: ID does not exist" containerID="bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264" Oct 31 00:57:00.483179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.483107 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264"} err="failed to get container status \"bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264\": rpc error: code = NotFound desc = could not find container \"bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264\": container with ID starting with bd0188905f6251e34e770a912f67a924dbd5d3fd725336ad5528aa002af94264 not found: ID does not exist" Oct 31 00:57:00.619569 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.619490 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c"] Oct 31 00:57:00.623493 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:00.623469 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3c636f-openshift-default-8685f58797-ph54c"] Oct 31 00:57:01.936035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:01.935994 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbfe5036-6f6e-49e3-9332-d558dd54b375" path="/var/lib/kubelet/pods/cbfe5036-6f6e-49e3-9332-d558dd54b375/volumes" Oct 31 00:57:01.936686 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:01.936666 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" path="/var/lib/kubelet/pods/de0b0377-cdf7-4f36-9af5-d5772a1dba9c/volumes" Oct 31 00:57:04.851545 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.851501 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5"] Oct 31 00:57:04.852247 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852228 5890 cpu_manager.go:414] "containerMap: removing container" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" Oct 31 00:57:04.852247 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852248 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" Oct 31 00:57:04.852386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852289 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e21029b5-0178-4c15-9013-61e8b08b7f90" containerName="payload-processing" Oct 31 00:57:04.852386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852295 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21029b5-0178-4c15-9013-61e8b08b7f90" containerName="payload-processing" Oct 31 00:57:04.852386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852319 5890 cpu_manager.go:414] "containerMap: removing container" podUID="cbfe5036-6f6e-49e3-9332-d558dd54b375" containerName="payload-pre-processing" Oct 31 00:57:04.852386 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852325 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfe5036-6f6e-49e3-9332-d558dd54b375" containerName="payload-pre-processing" Oct 31 00:57:04.852603 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852426 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfe5036-6f6e-49e3-9332-d558dd54b375" containerName="payload-pre-processing" Oct 31 00:57:04.852603 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852438 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" Oct 31 00:57:04.852603 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852449 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" Oct 31 00:57:04.852603 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.852455 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21029b5-0178-4c15-9013-61e8b08b7f90" containerName="payload-processing" Oct 31 00:57:04.858123 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.858095 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:04.911192 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.910145 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5"] Oct 31 00:57:04.951151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.951110 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f6979f\"" Oct 31 00:57:04.952643 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:04.951246 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f6979f-dockercfg-lgj2c\"" Oct 31 00:57:05.017815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.010351 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m"] Oct 31 00:57:05.017815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.011381 5890 cpu_manager.go:414] "containerMap: removing container" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" Oct 31 00:57:05.017815 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.011401 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0b0377-cdf7-4f36-9af5-d5772a1dba9c" containerName="istio-proxy" Oct 31 00:57:05.037807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.032194 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.051038 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.047964 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc55l\" (UniqueName: \"kubernetes.io/projected/25516389-22d4-4a4e-ad5c-914df4eeeba2-kube-api-access-mc55l\") pod \"payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:05.051038 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.048102 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25516389-22d4-4a4e-ad5c-914df4eeeba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:05.061814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.061595 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m"] Oct 31 00:57:05.151628 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.151264 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a8b36-54df-48ae-9351-3beb1afc67bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.151628 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.151425 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc55l\" (UniqueName: \"kubernetes.io/projected/25516389-22d4-4a4e-ad5c-914df4eeeba2-kube-api-access-mc55l\") pod \"payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:05.153068 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.152092 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25516389-22d4-4a4e-ad5c-914df4eeeba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:05.153068 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.152177 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52scn\" (UniqueName: \"kubernetes.io/projected/2c8a8b36-54df-48ae-9351-3beb1afc67bb-kube-api-access-52scn\") pod \"payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.153399 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.153370 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25516389-22d4-4a4e-ad5c-914df4eeeba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:05.210662 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.210630 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc55l\" (UniqueName: \"kubernetes.io/projected/25516389-22d4-4a4e-ad5c-914df4eeeba2-kube-api-access-mc55l\") pod \"payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:05.253694 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.253652 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a8b36-54df-48ae-9351-3beb1afc67bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.253887 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.253867 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52scn\" (UniqueName: \"kubernetes.io/projected/2c8a8b36-54df-48ae-9351-3beb1afc67bb-kube-api-access-52scn\") pod \"payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.254635 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.254599 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a8b36-54df-48ae-9351-3beb1afc67bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.313329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.313294 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52scn\" (UniqueName: \"kubernetes.io/projected/2c8a8b36-54df-48ae-9351-3beb1afc67bb-kube-api-access-52scn\") pod \"payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.403353 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.403267 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:05.497080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.497043 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:05.781406 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.781369 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m"] Oct 31 00:57:05.782977 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:57:05.782951 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8a8b36_54df_48ae_9351_3beb1afc67bb.slice/crio-f1434c94a8c72511bccdb7bc82c506956132a975c78d9c8d7d45ed883eed7da8 WatchSource:0}: Error finding container f1434c94a8c72511bccdb7bc82c506956132a975c78d9c8d7d45ed883eed7da8: Status 404 returned error can't find the container with id f1434c94a8c72511bccdb7bc82c506956132a975c78d9c8d7d45ed883eed7da8 Oct 31 00:57:05.894320 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:05.894292 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5"] Oct 31 00:57:05.894712 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:57:05.894689 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25516389_22d4_4a4e_ad5c_914df4eeeba2.slice/crio-272d288ff30190312069689bd9b933ecc27b5ca3b7e08a5a4b6452dbd30fd7c9 WatchSource:0}: Error finding container 272d288ff30190312069689bd9b933ecc27b5ca3b7e08a5a4b6452dbd30fd7c9: Status 404 returned error can't find the container with id 272d288ff30190312069689bd9b933ecc27b5ca3b7e08a5a4b6452dbd30fd7c9 Oct 31 00:57:06.517180 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.517137 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" event={"ID":"2c8a8b36-54df-48ae-9351-3beb1afc67bb","Type":"ContainerStarted","Data":"6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc"} Oct 31 00:57:06.517383 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.517206 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:06.517383 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.517232 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" event={"ID":"2c8a8b36-54df-48ae-9351-3beb1afc67bb","Type":"ContainerStarted","Data":"f1434c94a8c72511bccdb7bc82c506956132a975c78d9c8d7d45ed883eed7da8"} Oct 31 00:57:06.518839 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.518779 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" event={"ID":"25516389-22d4-4a4e-ad5c-914df4eeeba2","Type":"ContainerStarted","Data":"b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f"} Oct 31 00:57:06.518961 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.518859 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" event={"ID":"25516389-22d4-4a4e-ad5c-914df4eeeba2","Type":"ContainerStarted","Data":"272d288ff30190312069689bd9b933ecc27b5ca3b7e08a5a4b6452dbd30fd7c9"} Oct 31 00:57:06.518961 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.518912 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:06.631483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.631424 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" podStartSLOduration=2.631411411 podStartE2EDuration="2.631411411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:06.629226552 +0000 UTC m=+1718.831476986" watchObservedRunningTime="2026-10-31 00:57:06.631411411 +0000 UTC m=+1718.833661880" Oct 31 00:57:06.742400 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:06.742337 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" podStartSLOduration=2.742322985 podStartE2EDuration="2.742322985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:06.740833901 +0000 UTC m=+1718.943084338" watchObservedRunningTime="2026-10-31 00:57:06.742322985 +0000 UTC m=+1718.944573417" Oct 31 00:57:07.018928 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:57:07.018891 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:57:13.140702 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.140612 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d"] Oct 31 00:57:13.187226 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.187192 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d"] Oct 31 00:57:13.187547 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.187528 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.256832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.249430 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b647c3\"" Oct 31 00:57:13.256832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.250200 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b647c3-dockercfg-8zjxk\"" Oct 31 00:57:13.301575 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.294532 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69899ba-8421-46be-b145-9efa15ceb110-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.301575 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.294804 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhrr\" (UniqueName: \"kubernetes.io/projected/f69899ba-8421-46be-b145-9efa15ceb110-kube-api-access-lhhrr\") pod \"payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.396339 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.396242 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhrr\" (UniqueName: \"kubernetes.io/projected/f69899ba-8421-46be-b145-9efa15ceb110-kube-api-access-lhhrr\") pod \"payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.396506 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.396418 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69899ba-8421-46be-b145-9efa15ceb110-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.397319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.397293 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69899ba-8421-46be-b145-9efa15ceb110-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.518639 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.518600 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhrr\" (UniqueName: \"kubernetes.io/projected/f69899ba-8421-46be-b145-9efa15ceb110-kube-api-access-lhhrr\") pod \"payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.553694 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.553659 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:13.952979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:13.952824 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d"] Oct 31 00:57:13.959266 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:57:13.958541 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69899ba_8421_46be_b145_9efa15ceb110.slice/crio-9f5cc3a6d0d7cba9c36e22143211bdc401ba04e9671436069b8b2deff3f74d8c WatchSource:0}: Error finding container 9f5cc3a6d0d7cba9c36e22143211bdc401ba04e9671436069b8b2deff3f74d8c: Status 404 returned error can't find the container with id 9f5cc3a6d0d7cba9c36e22143211bdc401ba04e9671436069b8b2deff3f74d8c Oct 31 00:57:14.589209 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:14.589165 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" event={"ID":"f69899ba-8421-46be-b145-9efa15ceb110","Type":"ContainerStarted","Data":"8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741"} Oct 31 00:57:14.589648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:14.589238 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" event={"ID":"f69899ba-8421-46be-b145-9efa15ceb110","Type":"ContainerStarted","Data":"9f5cc3a6d0d7cba9c36e22143211bdc401ba04e9671436069b8b2deff3f74d8c"} Oct 31 00:57:14.589648 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:14.589269 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:14.708513 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:14.708378 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" podStartSLOduration=1.708357081 podStartE2EDuration="1.708357081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:14.700997699 +0000 UTC m=+1726.903248130" watchObservedRunningTime="2026-10-31 00:57:14.708357081 +0000 UTC m=+1726.910607514" Oct 31 00:57:15.127063 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.127026 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb"] Oct 31 00:57:15.131537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.131518 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.183217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.183186 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ac952a-dockercfg-g75xn\"" Oct 31 00:57:15.231092 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.231058 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ac952a\"" Oct 31 00:57:15.236852 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.236823 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb"] Oct 31 00:57:15.237014 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.236941 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b7291a-105a-4cee-9d55-a5ae80e1e571-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.237162 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.237146 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrl7\" (UniqueName: \"kubernetes.io/projected/b6b7291a-105a-4cee-9d55-a5ae80e1e571-kube-api-access-fkrl7\") pod \"payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.338538 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.338503 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrl7\" (UniqueName: \"kubernetes.io/projected/b6b7291a-105a-4cee-9d55-a5ae80e1e571-kube-api-access-fkrl7\") pod \"payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.338733 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.338612 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b7291a-105a-4cee-9d55-a5ae80e1e571-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.339414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.339387 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b7291a-105a-4cee-9d55-a5ae80e1e571-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.398143 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.397691 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrl7\" (UniqueName: \"kubernetes.io/projected/b6b7291a-105a-4cee-9d55-a5ae80e1e571-kube-api-access-fkrl7\") pod \"payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.447232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.447195 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:15.831658 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.831626 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb"] Oct 31 00:57:15.832221 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:57:15.832191 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b7291a_105a_4cee_9d55_a5ae80e1e571.slice/crio-fd567c3ffc3294e22661175be79b9f6f0d11f9e39d782a1e45734f68bc8155b2 WatchSource:0}: Error finding container fd567c3ffc3294e22661175be79b9f6f0d11f9e39d782a1e45734f68bc8155b2: Status 404 returned error can't find the container with id fd567c3ffc3294e22661175be79b9f6f0d11f9e39d782a1e45734f68bc8155b2 Oct 31 00:57:15.834588 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:15.834573 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:57:16.605864 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:16.605827 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" event={"ID":"b6b7291a-105a-4cee-9d55-a5ae80e1e571","Type":"ContainerStarted","Data":"fd567c3ffc3294e22661175be79b9f6f0d11f9e39d782a1e45734f68bc8155b2"} Oct 31 00:57:17.528972 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:17.528935 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:57:17.529354 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:17.529034 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:57:18.625873 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:18.625756 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" event={"ID":"b6b7291a-105a-4cee-9d55-a5ae80e1e571","Type":"ContainerStarted","Data":"91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899"} Oct 31 00:57:18.626311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:18.625910 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:18.739165 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:18.739093 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" podStartSLOduration=1.393666193 podStartE2EDuration="3.739069659s" totalImagesPullingTime="2.345403466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:18.738207767 +0000 UTC m=+1730.940458201" watchObservedRunningTime="2026-10-31 00:57:18.739069659 +0000 UTC m=+1730.941320094" Oct 31 00:57:25.601179 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:25.601142 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:57:27.719813 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:27.719745 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4"] Oct 31 00:57:29.635897 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:29.635859 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 00:57:37.023618 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:57:37.023572 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:57:42.741305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:42.741262 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" podUID="538a73cd-c247-4e5b-bd78-f1963689bb8b" containerName="main" containerID="cri-o://6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc" gracePeriod=45 Oct 31 00:57:43.016423 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.016392 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:57:43.113613 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.113578 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-tmp-dir\") pod \"538a73cd-c247-4e5b-bd78-f1963689bb8b\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " Oct 31 00:57:43.113826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.113669 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-dshm\") pod \"538a73cd-c247-4e5b-bd78-f1963689bb8b\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " Oct 31 00:57:43.113826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.113704 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538a73cd-c247-4e5b-bd78-f1963689bb8b-kube-api-access-htxrg\" (UniqueName: \"kubernetes.io/projected/538a73cd-c247-4e5b-bd78-f1963689bb8b-kube-api-access-htxrg\") pod \"538a73cd-c247-4e5b-bd78-f1963689bb8b\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " Oct 31 00:57:43.113826 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.113765 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-model-cache\") pod \"538a73cd-c247-4e5b-bd78-f1963689bb8b\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " Oct 31 00:57:43.114018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.113933 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-tmp-dir" pod "538a73cd-c247-4e5b-bd78-f1963689bb8b" (UID: "538a73cd-c247-4e5b-bd78-f1963689bb8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:43.114018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.113943 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/538a73cd-c247-4e5b-bd78-f1963689bb8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/538a73cd-c247-4e5b-bd78-f1963689bb8b-tls-certs\") pod \"538a73cd-c247-4e5b-bd78-f1963689bb8b\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " Oct 31 00:57:43.114135 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.114093 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-home\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-home\") pod \"538a73cd-c247-4e5b-bd78-f1963689bb8b\" (UID: \"538a73cd-c247-4e5b-bd78-f1963689bb8b\") " Oct 31 00:57:43.114239 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.114090 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-model-cache" pod "538a73cd-c247-4e5b-bd78-f1963689bb8b" (UID: "538a73cd-c247-4e5b-bd78-f1963689bb8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:43.114325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.114285 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-home" pod "538a73cd-c247-4e5b-bd78-f1963689bb8b" (UID: "538a73cd-c247-4e5b-bd78-f1963689bb8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:43.114829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.114770 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:43.114895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.114838 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:43.114895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.114854 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:43.116186 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.116163 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-dshm" pod "538a73cd-c247-4e5b-bd78-f1963689bb8b" (UID: "538a73cd-c247-4e5b-bd78-f1963689bb8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:43.116397 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.116374 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538a73cd-c247-4e5b-bd78-f1963689bb8b-kube-api-access-htxrg" pod "538a73cd-c247-4e5b-bd78-f1963689bb8b" (UID: "538a73cd-c247-4e5b-bd78-f1963689bb8b"). InnerVolumeSpecName "kube-api-access-htxrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:43.116521 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.116499 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538a73cd-c247-4e5b-bd78-f1963689bb8b-tls-certs" pod "538a73cd-c247-4e5b-bd78-f1963689bb8b" (UID: "538a73cd-c247-4e5b-bd78-f1963689bb8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:43.216361 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.216332 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538a73cd-c247-4e5b-bd78-f1963689bb8b-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:43.216361 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.216361 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a73cd-c247-4e5b-bd78-f1963689bb8b-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:43.216552 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.216371 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxrg\" (UniqueName: \"kubernetes.io/projected/538a73cd-c247-4e5b-bd78-f1963689bb8b-kube-api-access-htxrg\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:43.828535 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.828496 5890 generic.go:356] "Generic (PLEG): container finished" podID="538a73cd-c247-4e5b-bd78-f1963689bb8b" containerID="6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc" exitCode=0 Oct 31 00:57:43.829071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.828559 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" Oct 31 00:57:43.829071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.828574 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" event={"ID":"538a73cd-c247-4e5b-bd78-f1963689bb8b","Type":"ContainerDied","Data":"6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc"} Oct 31 00:57:43.829071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.828621 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4" event={"ID":"538a73cd-c247-4e5b-bd78-f1963689bb8b","Type":"ContainerDied","Data":"cf0ea44fa9328db1e8441ee8673bbbf7cb985f92e7f106ae73a03b4153d080d4"} Oct 31 00:57:43.829071 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.828638 5890 scope.go:122] "RemoveContainer" containerID="6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc" Oct 31 00:57:43.840209 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.840185 5890 scope.go:122] "RemoveContainer" containerID="6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc" Oct 31 00:57:43.840492 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:57:43.840473 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc\": container with ID starting with 6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc not found: ID does not exist" containerID="6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc" Oct 31 00:57:43.840545 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.840502 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc"} err="failed to get container status \"6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc\": rpc error: code = NotFound desc = could not find container \"6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc\": container with ID starting with 6369a8c485a4f429755418660ce3d1c2760877c4f8a434295efb18dfd0394ebc not found: ID does not exist" Oct 31 00:57:43.989008 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:43.988963 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4"] Oct 31 00:57:44.036216 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:44.036178 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7e6fe88b/e2e-lifecycle-model-7e6fe88b-kserve-796dfccd57-8wvn4"] Oct 31 00:57:45.934619 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:45.934586 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538a73cd-c247-4e5b-bd78-f1963689bb8b" path="/var/lib/kubelet/pods/538a73cd-c247-4e5b-bd78-f1963689bb8b/volumes" Oct 31 00:57:47.848524 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.848489 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5"] Oct 31 00:57:47.849216 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.849193 5890 cpu_manager.go:414] "containerMap: removing container" podUID="538a73cd-c247-4e5b-bd78-f1963689bb8b" containerName="main" Oct 31 00:57:47.849216 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.849216 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538a73cd-c247-4e5b-bd78-f1963689bb8b" containerName="main" Oct 31 00:57:47.849419 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.849404 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="538a73cd-c247-4e5b-bd78-f1963689bb8b" containerName="main" Oct 31 00:57:47.854515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.854492 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:47.902021 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.901984 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e2aba-5200-421c-80ba-d7e20a7ef582-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:47.902446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.902303 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b577d\" (UniqueName: \"kubernetes.io/projected/5d4e2aba-5200-421c-80ba-d7e20a7ef582-kube-api-access-b577d\") pod \"payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:47.907872 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.907844 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5"] Oct 31 00:57:47.950533 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.950500 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f66180\"" Oct 31 00:57:47.950966 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:47.950929 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f66180-dockercfg-jhlz6\"" Oct 31 00:57:48.003672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.003642 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b577d\" (UniqueName: \"kubernetes.io/projected/5d4e2aba-5200-421c-80ba-d7e20a7ef582-kube-api-access-b577d\") pod \"payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:48.003857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.003727 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e2aba-5200-421c-80ba-d7e20a7ef582-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:48.004425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.004404 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e2aba-5200-421c-80ba-d7e20a7ef582-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:48.063552 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.063513 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b577d\" (UniqueName: \"kubernetes.io/projected/5d4e2aba-5200-421c-80ba-d7e20a7ef582-kube-api-access-b577d\") pod \"payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:48.169754 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.169666 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:48.543779 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.543741 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5"] Oct 31 00:57:48.544801 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:57:48.544751 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4e2aba_5200_421c_80ba_d7e20a7ef582.slice/crio-6112e2df5f1e5129fbb420e36630398ac819170a37ca4968eda61bf6796442f6 WatchSource:0}: Error finding container 6112e2df5f1e5129fbb420e36630398ac819170a37ca4968eda61bf6796442f6: Status 404 returned error can't find the container with id 6112e2df5f1e5129fbb420e36630398ac819170a37ca4968eda61bf6796442f6 Oct 31 00:57:48.872745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.872653 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" event={"ID":"5d4e2aba-5200-421c-80ba-d7e20a7ef582","Type":"ContainerStarted","Data":"0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1"} Oct 31 00:57:48.872745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.872718 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" event={"ID":"5d4e2aba-5200-421c-80ba-d7e20a7ef582","Type":"ContainerStarted","Data":"6112e2df5f1e5129fbb420e36630398ac819170a37ca4968eda61bf6796442f6"} Oct 31 00:57:48.873246 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.872752 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:57:48.990221 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:48.990098 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" podStartSLOduration=1.990078153 podStartE2EDuration="1.990078153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:48.987354143 +0000 UTC m=+1761.189604577" watchObservedRunningTime="2026-10-31 00:57:48.990078153 +0000 UTC m=+1761.192328587" Oct 31 00:57:53.356126 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.356092 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp"] Oct 31 00:57:53.361857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.361832 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.459265 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.459236 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f6979f-serving-cert\"" Oct 31 00:57:53.463329 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.463302 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp"] Oct 31 00:57:53.485958 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.485925 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w4r\" (UniqueName: \"kubernetes.io/projected/195c7669-1d5e-413f-9827-d0ada1bf8e09-kube-api-access-r6w4r\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.486186 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.486166 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c7669-1d5e-413f-9827-d0ada1bf8e09-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.486307 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.486294 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.486505 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.486493 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-metrics-certs\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.588111 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.588076 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-metrics-certs\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.588315 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.588244 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w4r\" (UniqueName: \"kubernetes.io/projected/195c7669-1d5e-413f-9827-d0ada1bf8e09-kube-api-access-r6w4r\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.588315 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.588274 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c7669-1d5e-413f-9827-d0ada1bf8e09-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.588315 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.588311 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.589081 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.589053 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c7669-1d5e-413f-9827-d0ada1bf8e09-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.590651 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.590623 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.590778 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.590732 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-metrics-certs\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.648837 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.648739 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w4r\" (UniqueName: \"kubernetes.io/projected/195c7669-1d5e-413f-9827-d0ada1bf8e09-kube-api-access-r6w4r\") pod \"maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:53.678080 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:53.678050 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:54.066755 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:54.066721 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp"] Oct 31 00:57:54.067560 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:57:54.067533 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195c7669_1d5e_413f_9827_d0ada1bf8e09.slice/crio-d53e982e221679020bd13bb3cb3e6548ed1cb115a9cce0ff2ab056b9b74dd2eb WatchSource:0}: Error finding container d53e982e221679020bd13bb3cb3e6548ed1cb115a9cce0ff2ab056b9b74dd2eb: Status 404 returned error can't find the container with id d53e982e221679020bd13bb3cb3e6548ed1cb115a9cce0ff2ab056b9b74dd2eb Oct 31 00:57:54.919311 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:54.919276 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" event={"ID":"195c7669-1d5e-413f-9827-d0ada1bf8e09","Type":"ContainerStarted","Data":"d53e982e221679020bd13bb3cb3e6548ed1cb115a9cce0ff2ab056b9b74dd2eb"} Oct 31 00:57:55.934278 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:55.934248 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:57:55.934659 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:55.934298 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" event={"ID":"195c7669-1d5e-413f-9827-d0ada1bf8e09","Type":"ContainerStarted","Data":"8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d"} Oct 31 00:57:56.052867 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:56.052813 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" podStartSLOduration=1.997878144 podStartE2EDuration="3.05279697s" totalImagesPullingTime="1.054918826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:56.049173468 +0000 UTC m=+1768.251423904" watchObservedRunningTime="2026-10-31 00:57:56.05279697 +0000 UTC m=+1768.255047440" Oct 31 00:57:59.885314 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:57:59.885282 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:58:01.941483 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:01.941449 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 00:58:07.019772 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:58:07.019728 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:58:12.236206 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.236170 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss"] Oct 31 00:58:12.236981 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.236415 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" containerID="cri-o://6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34" gracePeriod=30 Oct 31 00:58:12.241744 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.241712 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7"] Oct 31 00:58:12.242005 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.241982 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" containerID="cri-o://ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c" gracePeriod=30 Oct 31 00:58:12.264274 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.264240 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx"] Oct 31 00:58:12.264514 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.264490 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" containerID="cri-o://b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd" gracePeriod=30 Oct 31 00:58:12.272480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.272454 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:58:12.272600 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:12.272534 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:58:13.564360 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:13.564315 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:58:13.564861 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:13.564377 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:58:13.569271 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:13.569242 5890 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:58:13.569427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:13.569288 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:58:17.576096 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.575693 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:58:17.623515 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.623486 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:58:17.627200 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.627180 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:58:17.688465 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688423 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-certs\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.688656 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688543 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-envoy\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.688656 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688595 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-credential-socket\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.688777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688662 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-podinfo\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.688777 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688708 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-socket\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.688898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688770 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-certs" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.688898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688808 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-token\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-token\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.688898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688842 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-kube-api-access-tcl77\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-kube-api-access-tcl77\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.689051 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688921 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-data\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.689051 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.688984 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istiod-ca-cert\") pod \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\" (UID: \"6e8e3c09-4b4d-499a-b38a-c5a18870579a\") " Oct 31 00:58:17.689142 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689087 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-credential-socket" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.689193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689124 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-socket" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.689296 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689278 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-data" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.689523 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689498 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istiod-ca-cert" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:17.689696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689560 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.689696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689577 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.689696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689591 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.689696 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.689603 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.691485 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.691461 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-envoy" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.691592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.691500 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-token" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.691592 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.691571 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-kube-api-access-tcl77" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "kube-api-access-tcl77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.691674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.691596 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-podinfo" pod "6e8e3c09-4b4d-499a-b38a-c5a18870579a" (UID: "6e8e3c09-4b4d-499a-b38a-c5a18870579a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790481 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-certs\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790536 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-certs\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790582 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-kube-api-access-wgmdm\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-kube-api-access-wgmdm\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790636 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-data\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790667 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-kube-api-access-7n7xt\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-kube-api-access-7n7xt\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790712 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-podinfo\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790753 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-credential-socket\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790850 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e88b93-8cef-471f-858b-45b842de5656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e88b93-8cef-471f-858b-45b842de5656-istiod-ca-cert\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790888 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-credential-socket\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790937 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-socket\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.790987 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-envoy\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.791031 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-token\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-token\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.791161 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-data\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.791203 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-envoy\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.791276 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-istio-token\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-istio-token\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.791315 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c97d1e-e95f-4828-b25c-181d42d3ea97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c97d1e-e95f-4828-b25c-181d42d3ea97-istiod-ca-cert\") pod \"88c97d1e-e95f-4828-b25c-181d42d3ea97\" (UID: \"88c97d1e-e95f-4828-b25c-181d42d3ea97\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.791348 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-socket\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.791395 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49e88b93-8cef-471f-858b-45b842de5656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e88b93-8cef-471f-858b-45b842de5656-istio-podinfo\") pod \"49e88b93-8cef-471f-858b-45b842de5656\" (UID: \"49e88b93-8cef-471f-858b-45b842de5656\") " Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.792582 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-data" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.792591 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-credential-socket" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.792830 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-socket" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.793062 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.793096 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.793113 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.793129 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcl77\" (UniqueName: \"kubernetes.io/projected/6e8e3c09-4b4d-499a-b38a-c5a18870579a-kube-api-access-tcl77\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.793144 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8e3c09-4b4d-499a-b38a-c5a18870579a-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.793766 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-credential-socket" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.793875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.793776 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-certs" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.795367 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.794123 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e88b93-8cef-471f-858b-45b842de5656-istiod-ca-cert" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:17.796223 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.795983 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c97d1e-e95f-4828-b25c-181d42d3ea97-istiod-ca-cert" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:17.796373 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.796311 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-certs" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.796530 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.796496 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-socket" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.796608 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.796534 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-data" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.798199 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.798173 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-kube-api-access-wgmdm" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "kube-api-access-wgmdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.798343 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.798323 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-istio-token" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.798415 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.798354 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-kube-api-access-7n7xt" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "kube-api-access-7n7xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.798480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.798414 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-envoy" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.798480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.798456 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-token" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.798480 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.798468 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-podinfo" pod "88c97d1e-e95f-4828-b25c-181d42d3ea97" (UID: "88c97d1e-e95f-4828-b25c-181d42d3ea97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:17.799073 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.799041 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49e88b93-8cef-471f-858b-45b842de5656-istio-podinfo" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:17.800316 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.800296 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-envoy" pod "49e88b93-8cef-471f-858b-45b842de5656" (UID: "49e88b93-8cef-471f-858b-45b842de5656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.894957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.894922 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e88b93-8cef-471f-858b-45b842de5656-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.894957 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.894956 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.894968 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.894983 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.894996 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895008 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895020 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895032 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895044 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c97d1e-e95f-4828-b25c-181d42d3ea97-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895056 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895070 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e88b93-8cef-471f-858b-45b842de5656-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895082 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c97d1e-e95f-4828-b25c-181d42d3ea97-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895100 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895113 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgmdm\" (UniqueName: \"kubernetes.io/projected/49e88b93-8cef-471f-858b-45b842de5656-kube-api-access-wgmdm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895127 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895138 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7xt\" (UniqueName: \"kubernetes.io/projected/88c97d1e-e95f-4828-b25c-181d42d3ea97-kube-api-access-7n7xt\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895152 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c97d1e-e95f-4828-b25c-181d42d3ea97-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.895215 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:17.895165 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e88b93-8cef-471f-858b-45b842de5656-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:18.100482 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.100387 5890 generic.go:356] "Generic (PLEG): container finished" podID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerID="6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34" exitCode=0 Oct 31 00:58:18.100727 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.100557 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" event={"ID":"6e8e3c09-4b4d-499a-b38a-c5a18870579a","Type":"ContainerDied","Data":"6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34"} Oct 31 00:58:18.100727 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.100613 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" event={"ID":"6e8e3c09-4b4d-499a-b38a-c5a18870579a","Type":"ContainerDied","Data":"7bb2fdef5f9d5d5285ec50c0b895e708d63b52b1235196a11b5dbff4b7d63bff"} Oct 31 00:58:18.100727 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.100643 5890 scope.go:122] "RemoveContainer" containerID="6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34" Oct 31 00:58:18.101443 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.101294 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss" Oct 31 00:58:18.109442 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.109407 5890 generic.go:356] "Generic (PLEG): container finished" podID="49e88b93-8cef-471f-858b-45b842de5656" containerID="ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c" exitCode=0 Oct 31 00:58:18.109603 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.109577 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" Oct 31 00:58:18.109713 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.109691 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" event={"ID":"49e88b93-8cef-471f-858b-45b842de5656","Type":"ContainerDied","Data":"ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c"} Oct 31 00:58:18.109829 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.109810 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7" event={"ID":"49e88b93-8cef-471f-858b-45b842de5656","Type":"ContainerDied","Data":"d8258e686126c8445c57dcd896e059af0aa0fdc89a94d3cc288817340a1e3b6b"} Oct 31 00:58:18.113778 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.113754 5890 generic.go:356] "Generic (PLEG): container finished" podID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerID="b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd" exitCode=0 Oct 31 00:58:18.113905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.113826 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" event={"ID":"88c97d1e-e95f-4828-b25c-181d42d3ea97","Type":"ContainerDied","Data":"b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd"} Oct 31 00:58:18.113905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.113846 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" Oct 31 00:58:18.113905 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.113870 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx" event={"ID":"88c97d1e-e95f-4828-b25c-181d42d3ea97","Type":"ContainerDied","Data":"6c637aa3de3adcf3dd0a22d94757c9ce4d1b4a0f50ea8d1791d19e682a993985"} Oct 31 00:58:18.123566 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.123539 5890 scope.go:122] "RemoveContainer" containerID="6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34" Oct 31 00:58:18.123938 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:58:18.123906 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34\": container with ID starting with 6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34 not found: ID does not exist" containerID="6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34" Oct 31 00:58:18.124043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.123946 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34"} err="failed to get container status \"6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34\": rpc error: code = NotFound desc = could not find container \"6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34\": container with ID starting with 6480912d2636cae84f64566ba833e5c12ac894a074de111e23d5367bb32b1c34 not found: ID does not exist" Oct 31 00:58:18.124043 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.123969 5890 scope.go:122] "RemoveContainer" containerID="ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c" Oct 31 00:58:18.136897 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.136861 5890 scope.go:122] "RemoveContainer" containerID="ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c" Oct 31 00:58:18.137278 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:58:18.137233 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c\": container with ID starting with ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c not found: ID does not exist" containerID="ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c" Oct 31 00:58:18.137411 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.137268 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c"} err="failed to get container status \"ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c\": rpc error: code = NotFound desc = could not find container \"ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c\": container with ID starting with ea348707885cf94f9e47938bf636613734abb8f36922897a5b35103fee5c888c not found: ID does not exist" Oct 31 00:58:18.137411 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.137293 5890 scope.go:122] "RemoveContainer" containerID="b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd" Oct 31 00:58:18.152013 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.151982 5890 scope.go:122] "RemoveContainer" containerID="b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd" Oct 31 00:58:18.153433 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:58:18.153405 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd\": container with ID starting with b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd not found: ID does not exist" containerID="b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd" Oct 31 00:58:18.153537 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.153442 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd"} err="failed to get container status \"b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd\": rpc error: code = NotFound desc = could not find container \"b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd\": container with ID starting with b4861c71338d96a7ff64e07a9b4d123105d50ca33dab1ddd8c6597d2e8fe77bd not found: ID does not exist" Oct 31 00:58:18.265277 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.265245 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7"] Oct 31 00:58:18.270714 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.270669 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-dj5n7"] Oct 31 00:58:18.375332 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.375253 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx"] Oct 31 00:58:18.379008 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.378974 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-7rxlx"] Oct 31 00:58:18.484757 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.484627 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss"] Oct 31 00:58:18.488607 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:18.488576 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-fhrss"] Oct 31 00:58:19.935018 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:19.934982 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e88b93-8cef-471f-858b-45b842de5656" path="/var/lib/kubelet/pods/49e88b93-8cef-471f-858b-45b842de5656/volumes" Oct 31 00:58:19.935643 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:19.935621 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" path="/var/lib/kubelet/pods/6e8e3c09-4b4d-499a-b38a-c5a18870579a/volumes" Oct 31 00:58:19.936224 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:19.936203 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" path="/var/lib/kubelet/pods/88c97d1e-e95f-4828-b25c-181d42d3ea97/volumes" Oct 31 00:58:22.517465 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.517426 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj"] Oct 31 00:58:22.519192 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519153 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" Oct 31 00:58:22.519192 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519183 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" Oct 31 00:58:22.519444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519227 5890 cpu_manager.go:414] "containerMap: removing container" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" Oct 31 00:58:22.519444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519235 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" Oct 31 00:58:22.519444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519308 5890 cpu_manager.go:414] "containerMap: removing container" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" Oct 31 00:58:22.519444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519316 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" Oct 31 00:58:22.519668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519461 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e88b93-8cef-471f-858b-45b842de5656" containerName="istio-proxy" Oct 31 00:58:22.519668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519474 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8e3c09-4b4d-499a-b38a-c5a18870579a" containerName="istio-proxy" Oct 31 00:58:22.519668 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.519490 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c97d1e-e95f-4828-b25c-181d42d3ea97" containerName="istio-proxy" Oct 31 00:58:22.524529 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.524500 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:22.554748 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.554696 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e37385-65a3-46b9-9389-b8525d69727a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:22.554942 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.554800 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdln\" (UniqueName: \"kubernetes.io/projected/25e37385-65a3-46b9-9389-b8525d69727a-kube-api-access-lxdln\") pod \"payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:22.579334 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.579300 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj"] Oct 31 00:58:22.622772 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.622729 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-eee7f1-dockercfg-7s9hk\"" Oct 31 00:58:22.622772 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.622747 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-eee7f1\"" Oct 31 00:58:22.655921 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.655885 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdln\" (UniqueName: \"kubernetes.io/projected/25e37385-65a3-46b9-9389-b8525d69727a-kube-api-access-lxdln\") pod \"payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:22.656196 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.656176 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e37385-65a3-46b9-9389-b8525d69727a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:22.656902 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.656884 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e37385-65a3-46b9-9389-b8525d69727a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:22.714697 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.714670 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdln\" (UniqueName: \"kubernetes.io/projected/25e37385-65a3-46b9-9389-b8525d69727a-kube-api-access-lxdln\") pod \"payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:22.840554 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:22.840474 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:23.217629 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:23.217599 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj"] Oct 31 00:58:23.218557 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:58:23.218532 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e37385_65a3_46b9_9389_b8525d69727a.slice/crio-f1681392701325232d306482053b71c0b49e6effb2cc7dbec6e35c2ffee12088 WatchSource:0}: Error finding container f1681392701325232d306482053b71c0b49e6effb2cc7dbec6e35c2ffee12088: Status 404 returned error can't find the container with id f1681392701325232d306482053b71c0b49e6effb2cc7dbec6e35c2ffee12088 Oct 31 00:58:24.167338 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:24.167303 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" event={"ID":"25e37385-65a3-46b9-9389-b8525d69727a","Type":"ContainerStarted","Data":"8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8"} Oct 31 00:58:24.167865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:24.167364 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" event={"ID":"25e37385-65a3-46b9-9389-b8525d69727a","Type":"ContainerStarted","Data":"f1681392701325232d306482053b71c0b49e6effb2cc7dbec6e35c2ffee12088"} Oct 31 00:58:24.167865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:24.167393 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:24.284360 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:24.284306 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" podStartSLOduration=2.284292203 podStartE2EDuration="2.284292203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:24.282942211 +0000 UTC m=+1796.485192662" watchObservedRunningTime="2026-10-31 00:58:24.284292203 +0000 UTC m=+1796.486542614" Oct 31 00:58:28.195077 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:28.195042 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf"] Oct 31 00:58:28.725620 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:28.725591 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:58:28.841408 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:28.841381 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 00:58:33.197061 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.197017 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" podUID="6bce1d06-269d-4bdb-90bc-0e07f11a164e" containerName="payload-pre-processing" containerID="cri-o://ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8" gracePeriod=25 Oct 31 00:58:33.506752 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.506721 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:58:33.601411 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.601375 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bce1d06-269d-4bdb-90bc-0e07f11a164e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce1d06-269d-4bdb-90bc-0e07f11a164e-plugins-config-volume\") pod \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " Oct 31 00:58:33.601585 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.601426 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bce1d06-269d-4bdb-90bc-0e07f11a164e-kube-api-access-k85ws\" (UniqueName: \"kubernetes.io/projected/6bce1d06-269d-4bdb-90bc-0e07f11a164e-kube-api-access-k85ws\") pod \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\" (UID: \"6bce1d06-269d-4bdb-90bc-0e07f11a164e\") " Oct 31 00:58:33.601855 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.601830 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bce1d06-269d-4bdb-90bc-0e07f11a164e-plugins-config-volume" pod "6bce1d06-269d-4bdb-90bc-0e07f11a164e" (UID: "6bce1d06-269d-4bdb-90bc-0e07f11a164e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:33.601989 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.601971 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce1d06-269d-4bdb-90bc-0e07f11a164e-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:33.603604 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.603581 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bce1d06-269d-4bdb-90bc-0e07f11a164e-kube-api-access-k85ws" pod "6bce1d06-269d-4bdb-90bc-0e07f11a164e" (UID: "6bce1d06-269d-4bdb-90bc-0e07f11a164e"). InnerVolumeSpecName "kube-api-access-k85ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:33.703410 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:33.703338 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k85ws\" (UniqueName: \"kubernetes.io/projected/6bce1d06-269d-4bdb-90bc-0e07f11a164e-kube-api-access-k85ws\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:34.252140 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.252107 5890 generic.go:356] "Generic (PLEG): container finished" podID="6bce1d06-269d-4bdb-90bc-0e07f11a164e" containerID="ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8" exitCode=0 Oct 31 00:58:34.252621 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.252149 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" Oct 31 00:58:34.252621 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.252192 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" event={"ID":"6bce1d06-269d-4bdb-90bc-0e07f11a164e","Type":"ContainerDied","Data":"ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8"} Oct 31 00:58:34.252621 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.252247 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf" event={"ID":"6bce1d06-269d-4bdb-90bc-0e07f11a164e","Type":"ContainerDied","Data":"10bf95891264a30a90b3ebf3de7859423fc4be2245705d7d248eaac1f39336ad"} Oct 31 00:58:34.252621 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.252275 5890 scope.go:122] "RemoveContainer" containerID="ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8" Oct 31 00:58:34.265138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.265117 5890 scope.go:122] "RemoveContainer" containerID="ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8" Oct 31 00:58:34.265441 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:58:34.265414 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8\": container with ID starting with ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8 not found: ID does not exist" containerID="ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8" Oct 31 00:58:34.265503 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.265451 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8"} err="failed to get container status \"ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8\": rpc error: code = NotFound desc = could not find container \"ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8\": container with ID starting with ed3c823ab09268068a6490d8d5440550ca9690e906321bbf6463b0c0e6171cb8 not found: ID does not exist" Oct 31 00:58:34.416193 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.416157 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf"] Oct 31 00:58:34.419352 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:34.419324 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-6c56c76d59-mn7lf"] Oct 31 00:58:35.181577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:35.181542 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 00:58:35.934672 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:58:35.934637 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bce1d06-269d-4bdb-90bc-0e07f11a164e" path="/var/lib/kubelet/pods/6bce1d06-269d-4bdb-90bc-0e07f11a164e/volumes" Oct 31 00:58:37.020008 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:58:37.019967 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:59:01.545035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:01.545000 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f"] Oct 31 00:59:01.545566 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:01.545225 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" containerID="cri-o://27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa" gracePeriod=30 Oct 31 00:59:06.602704 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.602666 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m"] Oct 31 00:59:06.650035 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.650000 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5"] Oct 31 00:59:06.849319 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.849290 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:59:06.983817 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.983746 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-credential-socket\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.985536 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-kube-api-access-zd55p\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-kube-api-access-zd55p\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.985679 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-socket\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.985729 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-data\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.985818 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-envoy\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.985890 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-token\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-token\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.985930 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istiod-ca-cert\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.986068 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-podinfo\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.987384 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.986114 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-certs\") pod \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\" (UID: \"73d6e5c8-761b-46c9-9ab3-eefd2844d806\") " Oct 31 00:59:06.988237 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.987994 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-certs" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:06.988745 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.988386 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-credential-socket" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:06.991993 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.991698 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-kube-api-access-zd55p" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "kube-api-access-zd55p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:07.002626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.994155 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istiod-ca-cert" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:07.002626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.995539 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-socket" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:07.002626 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:06.995716 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-data" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:07.007382 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.007272 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-token" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:07.007704 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.007676 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-envoy" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:07.013098 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.013067 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-podinfo" pod "73d6e5c8-761b-46c9-9ab3-eefd2844d806" (UID: "73d6e5c8-761b-46c9-9ab3-eefd2844d806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:59:07.041069 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:07.041034 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:59:07.089102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.088943 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.088999 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd55p\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-kube-api-access-zd55p\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.089017 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089519 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.089481 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089519 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.089509 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.089532 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.089548 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.089564 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d6e5c8-761b-46c9-9ab3-eefd2844d806-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.089708 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.089578 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d6e5c8-761b-46c9-9ab3-eefd2844d806-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:07.525385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.525349 5890 generic.go:356] "Generic (PLEG): container finished" podID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerID="27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa" exitCode=0 Oct 31 00:59:07.525580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.525400 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" Oct 31 00:59:07.525580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.525441 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" event={"ID":"73d6e5c8-761b-46c9-9ab3-eefd2844d806","Type":"ContainerDied","Data":"27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa"} Oct 31 00:59:07.525580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.525505 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f" event={"ID":"73d6e5c8-761b-46c9-9ab3-eefd2844d806","Type":"ContainerDied","Data":"df37b30541675a518a4e7276bfe651de7545ac55a4d1ee124d15ece331d8104f"} Oct 31 00:59:07.525580 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.525530 5890 scope.go:122] "RemoveContainer" containerID="27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa" Oct 31 00:59:07.536278 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.536259 5890 scope.go:122] "RemoveContainer" containerID="f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381" Oct 31 00:59:07.549217 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.549199 5890 scope.go:122] "RemoveContainer" containerID="27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa" Oct 31 00:59:07.549515 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:07.549495 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa\": container with ID starting with 27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa not found: ID does not exist" containerID="27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa" Oct 31 00:59:07.549573 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.549526 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa"} err="failed to get container status \"27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa\": rpc error: code = NotFound desc = could not find container \"27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa\": container with ID starting with 27f332cba5a1348ac4e6f1d41376cf61889035186da2481997d9371e44469faa not found: ID does not exist" Oct 31 00:59:07.549573 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.549545 5890 scope.go:122] "RemoveContainer" containerID="f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381" Oct 31 00:59:07.549802 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:07.549766 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381\": container with ID starting with f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381 not found: ID does not exist" containerID="f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381" Oct 31 00:59:07.549856 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.549802 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381"} err="failed to get container status \"f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381\": rpc error: code = NotFound desc = could not find container \"f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381\": container with ID starting with f8decf9750d4239ebfcb8767001035747b2c6b08b5c824638358b4a1de3d8381 not found: ID does not exist" Oct 31 00:59:07.688988 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.688951 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f"] Oct 31 00:59:07.692929 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.692901 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e942ab-openshift-default-9c697d45-s727f"] Oct 31 00:59:07.935427 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:07.935394 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" path="/var/lib/kubelet/pods/73d6e5c8-761b-46c9-9ab3-eefd2844d806/volumes" Oct 31 00:59:11.603907 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:11.603859 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" podUID="2c8a8b36-54df-48ae-9351-3beb1afc67bb" containerName="payload-processing" containerID="cri-o://6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc" gracePeriod=25 Oct 31 00:59:11.650551 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:11.650510 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" podUID="25516389-22d4-4a4e-ad5c-914df4eeeba2" containerName="payload-pre-processing" containerID="cri-o://b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f" gracePeriod=25 Oct 31 00:59:11.974832 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:11.974745 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:59:12.014860 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.014823 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:59:12.055175 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.055144 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8a8b36-54df-48ae-9351-3beb1afc67bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a8b36-54df-48ae-9351-3beb1afc67bb-plugins-config-volume\") pod \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " Oct 31 00:59:12.055405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.055384 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25516389-22d4-4a4e-ad5c-914df4eeeba2-kube-api-access-mc55l\" (UniqueName: \"kubernetes.io/projected/25516389-22d4-4a4e-ad5c-914df4eeeba2-kube-api-access-mc55l\") pod \"25516389-22d4-4a4e-ad5c-914df4eeeba2\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " Oct 31 00:59:12.055557 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.055543 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8a8b36-54df-48ae-9351-3beb1afc67bb-kube-api-access-52scn\" (UniqueName: \"kubernetes.io/projected/2c8a8b36-54df-48ae-9351-3beb1afc67bb-kube-api-access-52scn\") pod \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\" (UID: \"2c8a8b36-54df-48ae-9351-3beb1afc67bb\") " Oct 31 00:59:12.055763 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.055734 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25516389-22d4-4a4e-ad5c-914df4eeeba2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25516389-22d4-4a4e-ad5c-914df4eeeba2-plugins-config-volume\") pod \"25516389-22d4-4a4e-ad5c-914df4eeeba2\" (UID: \"25516389-22d4-4a4e-ad5c-914df4eeeba2\") " Oct 31 00:59:12.057293 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.056985 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8a8b36-54df-48ae-9351-3beb1afc67bb-plugins-config-volume" pod "2c8a8b36-54df-48ae-9351-3beb1afc67bb" (UID: "2c8a8b36-54df-48ae-9351-3beb1afc67bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:12.057906 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.057884 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25516389-22d4-4a4e-ad5c-914df4eeeba2-plugins-config-volume" pod "25516389-22d4-4a4e-ad5c-914df4eeeba2" (UID: "25516389-22d4-4a4e-ad5c-914df4eeeba2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:12.059340 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.059314 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8a8b36-54df-48ae-9351-3beb1afc67bb-kube-api-access-52scn" pod "2c8a8b36-54df-48ae-9351-3beb1afc67bb" (UID: "2c8a8b36-54df-48ae-9351-3beb1afc67bb"). InnerVolumeSpecName "kube-api-access-52scn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:12.060276 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.060255 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25516389-22d4-4a4e-ad5c-914df4eeeba2-kube-api-access-mc55l" pod "25516389-22d4-4a4e-ad5c-914df4eeeba2" (UID: "25516389-22d4-4a4e-ad5c-914df4eeeba2"). InnerVolumeSpecName "kube-api-access-mc55l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:12.203861 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.203678 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8a8b36-54df-48ae-9351-3beb1afc67bb-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:12.203861 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.203715 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc55l\" (UniqueName: \"kubernetes.io/projected/25516389-22d4-4a4e-ad5c-914df4eeeba2-kube-api-access-mc55l\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:12.203861 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.203730 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52scn\" (UniqueName: \"kubernetes.io/projected/2c8a8b36-54df-48ae-9351-3beb1afc67bb-kube-api-access-52scn\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:12.203861 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.203745 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25516389-22d4-4a4e-ad5c-914df4eeeba2-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:12.562325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.562225 5890 generic.go:356] "Generic (PLEG): container finished" podID="2c8a8b36-54df-48ae-9351-3beb1afc67bb" containerID="6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc" exitCode=0 Oct 31 00:59:12.562325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.562281 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" Oct 31 00:59:12.562325 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.562304 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" event={"ID":"2c8a8b36-54df-48ae-9351-3beb1afc67bb","Type":"ContainerDied","Data":"6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc"} Oct 31 00:59:12.562599 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.562365 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m" event={"ID":"2c8a8b36-54df-48ae-9351-3beb1afc67bb","Type":"ContainerDied","Data":"f1434c94a8c72511bccdb7bc82c506956132a975c78d9c8d7d45ed883eed7da8"} Oct 31 00:59:12.562599 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.562389 5890 scope.go:122] "RemoveContainer" containerID="6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc" Oct 31 00:59:12.564072 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.564034 5890 generic.go:356] "Generic (PLEG): container finished" podID="25516389-22d4-4a4e-ad5c-914df4eeeba2" containerID="b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f" exitCode=0 Oct 31 00:59:12.564150 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.564069 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" event={"ID":"25516389-22d4-4a4e-ad5c-914df4eeeba2","Type":"ContainerDied","Data":"b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f"} Oct 31 00:59:12.564150 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.564083 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" Oct 31 00:59:12.564150 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.564109 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5" event={"ID":"25516389-22d4-4a4e-ad5c-914df4eeeba2","Type":"ContainerDied","Data":"272d288ff30190312069689bd9b933ecc27b5ca3b7e08a5a4b6452dbd30fd7c9"} Oct 31 00:59:12.576511 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.576485 5890 scope.go:122] "RemoveContainer" containerID="6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc" Oct 31 00:59:12.576879 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:12.576850 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc\": container with ID starting with 6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc not found: ID does not exist" containerID="6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc" Oct 31 00:59:12.576979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.576887 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc"} err="failed to get container status \"6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc\": rpc error: code = NotFound desc = could not find container \"6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc\": container with ID starting with 6c0a141f4c7c36f1160464e5ddf1478278f18b9c8529d8849d4f154e7e5ae8dc not found: ID does not exist" Oct 31 00:59:12.576979 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.576910 5890 scope.go:122] "RemoveContainer" containerID="b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f" Oct 31 00:59:12.592189 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.592171 5890 scope.go:122] "RemoveContainer" containerID="b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f" Oct 31 00:59:12.592491 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:12.592472 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f\": container with ID starting with b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f not found: ID does not exist" containerID="b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f" Oct 31 00:59:12.592567 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.592498 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f"} err="failed to get container status \"b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f\": rpc error: code = NotFound desc = could not find container \"b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f\": container with ID starting with b34837acdfdd1d7aba15c014dacf66d6d810814923884e5e14e0bac1d89ee60f not found: ID does not exist" Oct 31 00:59:12.725851 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.725810 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m"] Oct 31 00:59:12.732596 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.732565 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f6979f-65c8bd494f-hbt4m"] Oct 31 00:59:12.840705 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.840613 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5"] Oct 31 00:59:12.843601 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:12.843573 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f6979f-f774f5b55-n5nw5"] Oct 31 00:59:13.782467 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.782417 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr"] Oct 31 00:59:13.783237 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783214 5890 cpu_manager.go:414] "containerMap: removing container" podUID="25516389-22d4-4a4e-ad5c-914df4eeeba2" containerName="payload-pre-processing" Oct 31 00:59:13.783237 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783235 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25516389-22d4-4a4e-ad5c-914df4eeeba2" containerName="payload-pre-processing" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783249 5890 cpu_manager.go:414] "containerMap: removing container" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783254 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783276 5890 cpu_manager.go:414] "containerMap: removing container" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783281 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783293 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6bce1d06-269d-4bdb-90bc-0e07f11a164e" containerName="payload-pre-processing" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783298 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bce1d06-269d-4bdb-90bc-0e07f11a164e" containerName="payload-pre-processing" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783321 5890 cpu_manager.go:414] "containerMap: removing container" podUID="2c8a8b36-54df-48ae-9351-3beb1afc67bb" containerName="payload-processing" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783326 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8a8b36-54df-48ae-9351-3beb1afc67bb" containerName="payload-processing" Oct 31 00:59:13.783414 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783415 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8a8b36-54df-48ae-9351-3beb1afc67bb" containerName="payload-processing" Oct 31 00:59:13.783895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783424 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="25516389-22d4-4a4e-ad5c-914df4eeeba2" containerName="payload-pre-processing" Oct 31 00:59:13.783895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783432 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" Oct 31 00:59:13.783895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783441 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d6e5c8-761b-46c9-9ab3-eefd2844d806" containerName="istio-proxy" Oct 31 00:59:13.783895 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.783447 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bce1d06-269d-4bdb-90bc-0e07f11a164e" containerName="payload-pre-processing" Oct 31 00:59:13.788586 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.788564 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:13.843910 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.843880 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr"] Oct 31 00:59:13.914814 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.911330 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8c08e7-dockercfg-fp8ln\"" Oct 31 00:59:13.918857 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.913679 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8c08e7\"" Oct 31 00:59:13.965880 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.953682 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cebe79fe-5fa3-49c5-a490-f640f57fe999-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:13.966310 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:13.966283 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpvlj\" (UniqueName: \"kubernetes.io/projected/cebe79fe-5fa3-49c5-a490-f640f57fe999-kube-api-access-xpvlj\") pod \"payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:14.008006 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.007962 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25516389-22d4-4a4e-ad5c-914df4eeeba2" path="/var/lib/kubelet/pods/25516389-22d4-4a4e-ad5c-914df4eeeba2/volumes" Oct 31 00:59:14.008636 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.008615 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8a8b36-54df-48ae-9351-3beb1afc67bb" path="/var/lib/kubelet/pods/2c8a8b36-54df-48ae-9351-3beb1afc67bb/volumes" Oct 31 00:59:14.067671 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.067587 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpvlj\" (UniqueName: \"kubernetes.io/projected/cebe79fe-5fa3-49c5-a490-f640f57fe999-kube-api-access-xpvlj\") pod \"payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:14.067838 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.067760 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cebe79fe-5fa3-49c5-a490-f640f57fe999-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:14.068634 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.068602 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cebe79fe-5fa3-49c5-a490-f640f57fe999-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:14.126850 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.126817 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpvlj\" (UniqueName: \"kubernetes.io/projected/cebe79fe-5fa3-49c5-a490-f640f57fe999-kube-api-access-xpvlj\") pod \"payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:14.407856 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.406046 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:14.554151 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.554113 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d"] Oct 31 00:59:14.842898 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:14.842861 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr"] Oct 31 00:59:14.843479 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:59:14.843455 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebe79fe_5fa3_49c5_a490_f640f57fe999.slice/crio-3b19f34f6f54e567ebd9555bb3c38c0e0108acc8d80ec44ec1121650e66a57de WatchSource:0}: Error finding container 3b19f34f6f54e567ebd9555bb3c38c0e0108acc8d80ec44ec1121650e66a57de: Status 404 returned error can't find the container with id 3b19f34f6f54e567ebd9555bb3c38c0e0108acc8d80ec44ec1121650e66a57de Oct 31 00:59:15.595600 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:15.595561 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" event={"ID":"cebe79fe-5fa3-49c5-a490-f640f57fe999","Type":"ContainerStarted","Data":"cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8"} Oct 31 00:59:15.595824 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:15.595696 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:15.595824 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:15.595732 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" event={"ID":"cebe79fe-5fa3-49c5-a490-f640f57fe999","Type":"ContainerStarted","Data":"3b19f34f6f54e567ebd9555bb3c38c0e0108acc8d80ec44ec1121650e66a57de"} Oct 31 00:59:15.710444 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:15.710382 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" podStartSLOduration=2.710367265 podStartE2EDuration="2.710367265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:15.707266793 +0000 UTC m=+1847.909517226" watchObservedRunningTime="2026-10-31 00:59:15.710367265 +0000 UTC m=+1847.912617717" Oct 31 00:59:19.555517 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.555439 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" podUID="f69899ba-8421-46be-b145-9efa15ceb110" containerName="payload-processing" containerID="cri-o://8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741" gracePeriod=25 Oct 31 00:59:19.831470 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.831441 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:59:19.882398 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.882364 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69899ba-8421-46be-b145-9efa15ceb110-kube-api-access-lhhrr\" (UniqueName: \"kubernetes.io/projected/f69899ba-8421-46be-b145-9efa15ceb110-kube-api-access-lhhrr\") pod \"f69899ba-8421-46be-b145-9efa15ceb110\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " Oct 31 00:59:19.882587 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.882433 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69899ba-8421-46be-b145-9efa15ceb110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69899ba-8421-46be-b145-9efa15ceb110-plugins-config-volume\") pod \"f69899ba-8421-46be-b145-9efa15ceb110\" (UID: \"f69899ba-8421-46be-b145-9efa15ceb110\") " Oct 31 00:59:19.882862 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.882836 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69899ba-8421-46be-b145-9efa15ceb110-plugins-config-volume" pod "f69899ba-8421-46be-b145-9efa15ceb110" (UID: "f69899ba-8421-46be-b145-9efa15ceb110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:19.882964 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.882919 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69899ba-8421-46be-b145-9efa15ceb110-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:19.885102 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.885074 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69899ba-8421-46be-b145-9efa15ceb110-kube-api-access-lhhrr" pod "f69899ba-8421-46be-b145-9efa15ceb110" (UID: "f69899ba-8421-46be-b145-9efa15ceb110"). InnerVolumeSpecName "kube-api-access-lhhrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:19.987396 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:19.987097 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhhrr\" (UniqueName: \"kubernetes.io/projected/f69899ba-8421-46be-b145-9efa15ceb110-kube-api-access-lhhrr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:20.633721 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.633687 5890 generic.go:356] "Generic (PLEG): container finished" podID="f69899ba-8421-46be-b145-9efa15ceb110" containerID="8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741" exitCode=0 Oct 31 00:59:20.634176 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.633745 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" Oct 31 00:59:20.634176 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.633769 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" event={"ID":"f69899ba-8421-46be-b145-9efa15ceb110","Type":"ContainerDied","Data":"8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741"} Oct 31 00:59:20.634176 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.633837 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d" event={"ID":"f69899ba-8421-46be-b145-9efa15ceb110","Type":"ContainerDied","Data":"9f5cc3a6d0d7cba9c36e22143211bdc401ba04e9671436069b8b2deff3f74d8c"} Oct 31 00:59:20.634176 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.633866 5890 scope.go:122] "RemoveContainer" containerID="8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741" Oct 31 00:59:20.654635 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.654605 5890 scope.go:122] "RemoveContainer" containerID="8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741" Oct 31 00:59:20.655025 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:20.654996 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741\": container with ID starting with 8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741 not found: ID does not exist" containerID="8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741" Oct 31 00:59:20.655172 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.655033 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741"} err="failed to get container status \"8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741\": rpc error: code = NotFound desc = could not find container \"8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741\": container with ID starting with 8c63d3de0ef42c12cad3816e7039331500ac63f0eb7135965c866815cfff6741 not found: ID does not exist" Oct 31 00:59:20.795974 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.795934 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d"] Oct 31 00:59:20.842699 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:20.842666 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b647c3-5b5b948867-wzn9d"] Oct 31 00:59:21.728574 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.728539 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n"] Oct 31 00:59:21.729405 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.729382 5890 cpu_manager.go:414] "containerMap: removing container" podUID="f69899ba-8421-46be-b145-9efa15ceb110" containerName="payload-processing" Oct 31 00:59:21.729460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.729408 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69899ba-8421-46be-b145-9efa15ceb110" containerName="payload-processing" Oct 31 00:59:21.729603 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.729588 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69899ba-8421-46be-b145-9efa15ceb110" containerName="payload-processing" Oct 31 00:59:21.734428 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.734408 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:21.785110 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.785080 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eb41d3-dockercfg-kwm42\"" Oct 31 00:59:21.785433 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.785414 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eb41d3\"" Oct 31 00:59:21.787875 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.787840 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n"] Oct 31 00:59:21.813489 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.813451 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpzx\" (UniqueName: \"kubernetes.io/projected/a7145009-3452-4f30-b2e5-6cae7c650b8b-kube-api-access-kdpzx\") pod \"payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:21.813655 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.813513 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7145009-3452-4f30-b2e5-6cae7c650b8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:21.915473 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.915432 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x"] Oct 31 00:59:21.918764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.916595 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpzx\" (UniqueName: \"kubernetes.io/projected/a7145009-3452-4f30-b2e5-6cae7c650b8b-kube-api-access-kdpzx\") pod \"payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:21.919160 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.919135 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7145009-3452-4f30-b2e5-6cae7c650b8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:21.920339 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.920316 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7145009-3452-4f30-b2e5-6cae7c650b8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:21.927316 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.927291 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:21.956023 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.955989 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69899ba-8421-46be-b145-9efa15ceb110" path="/var/lib/kubelet/pods/f69899ba-8421-46be-b145-9efa15ceb110/volumes" Oct 31 00:59:21.956479 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.956457 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x"] Oct 31 00:59:21.977865 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:21.977834 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpzx\" (UniqueName: \"kubernetes.io/projected/a7145009-3452-4f30-b2e5-6cae7c650b8b-kube-api-access-kdpzx\") pod \"payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:22.020675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.020594 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwg2b\" (UniqueName: \"kubernetes.io/projected/0562d880-8f7f-4673-9125-273738a1a76f-kube-api-access-rwg2b\") pod \"payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:22.020675 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.020652 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0562d880-8f7f-4673-9125-273738a1a76f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:22.048375 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.048350 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:22.125450 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.122450 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwg2b\" (UniqueName: \"kubernetes.io/projected/0562d880-8f7f-4673-9125-273738a1a76f-kube-api-access-rwg2b\") pod \"payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:22.125450 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.122513 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0562d880-8f7f-4673-9125-273738a1a76f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:22.125450 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.123595 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0562d880-8f7f-4673-9125-273738a1a76f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:22.182807 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.182548 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwg2b\" (UniqueName: \"kubernetes.io/projected/0562d880-8f7f-4673-9125-273738a1a76f-kube-api-access-rwg2b\") pod \"payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:22.257496 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.257464 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:22.448446 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:59:22.447930 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7145009_3452_4f30_b2e5_6cae7c650b8b.slice/crio-3442deea443851e02350977f4bb30c8ad0dee9d044449938b10aab7d1a4035b5 WatchSource:0}: Error finding container 3442deea443851e02350977f4bb30c8ad0dee9d044449938b10aab7d1a4035b5: Status 404 returned error can't find the container with id 3442deea443851e02350977f4bb30c8ad0dee9d044449938b10aab7d1a4035b5 Oct 31 00:59:22.448658 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.448599 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n"] Oct 31 00:59:22.631113 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.631064 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x"] Oct 31 00:59:22.632248 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:59:22.632217 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0562d880_8f7f_4673_9125_273738a1a76f.slice/crio-3eae4dd0172f18a0f8c1a83d4cf0086811cdc7b6c922ca0f6d69b42565e2544e WatchSource:0}: Error finding container 3eae4dd0172f18a0f8c1a83d4cf0086811cdc7b6c922ca0f6d69b42565e2544e: Status 404 returned error can't find the container with id 3eae4dd0172f18a0f8c1a83d4cf0086811cdc7b6c922ca0f6d69b42565e2544e Oct 31 00:59:22.651561 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.651536 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" event={"ID":"0562d880-8f7f-4673-9125-273738a1a76f","Type":"ContainerStarted","Data":"3eae4dd0172f18a0f8c1a83d4cf0086811cdc7b6c922ca0f6d69b42565e2544e"} Oct 31 00:59:22.653307 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.653279 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" event={"ID":"a7145009-3452-4f30-b2e5-6cae7c650b8b","Type":"ContainerStarted","Data":"9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298"} Oct 31 00:59:22.653436 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.653332 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" event={"ID":"a7145009-3452-4f30-b2e5-6cae7c650b8b","Type":"ContainerStarted","Data":"3442deea443851e02350977f4bb30c8ad0dee9d044449938b10aab7d1a4035b5"} Oct 31 00:59:22.653436 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.653365 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:22.772952 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:22.772888 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" podStartSLOduration=1.7728703700000001 podStartE2EDuration="1.77287037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:22.76840262 +0000 UTC m=+1854.970653054" watchObservedRunningTime="2026-10-31 00:59:22.77287037 +0000 UTC m=+1854.975120803" Oct 31 00:59:23.663587 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:23.663535 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" event={"ID":"0562d880-8f7f-4673-9125-273738a1a76f","Type":"ContainerStarted","Data":"5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b"} Oct 31 00:59:23.663958 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:23.663933 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:23.777728 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:23.777673 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" podStartSLOduration=2.777659137 podStartE2EDuration="2.777659137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:23.775306841 +0000 UTC m=+1855.977557277" watchObservedRunningTime="2026-10-31 00:59:23.777659137 +0000 UTC m=+1855.979909570" Oct 31 00:59:26.607173 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:26.607144 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 00:59:33.405059 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.405020 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg"] Oct 31 00:59:33.410698 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.410681 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:33.465930 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.465896 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg"] Oct 31 00:59:33.476632 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.476590 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3235a9-ace9-49c7-ae0a-5567ccacc120-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:33.476837 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.476687 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdj54\" (UniqueName: \"kubernetes.io/projected/2d3235a9-ace9-49c7-ae0a-5567ccacc120-kube-api-access-gdj54\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:33.508232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.508197 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cb2b8a\"" Oct 31 00:59:33.508412 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.508301 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cb2b8a-dockercfg-lstdr\"" Oct 31 00:59:33.577766 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.577732 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3235a9-ace9-49c7-ae0a-5567ccacc120-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:33.577960 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.577836 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdj54\" (UniqueName: \"kubernetes.io/projected/2d3235a9-ace9-49c7-ae0a-5567ccacc120-kube-api-access-gdj54\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:33.578665 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.578642 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3235a9-ace9-49c7-ae0a-5567ccacc120-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:33.637344 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.637318 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdj54\" (UniqueName: \"kubernetes.io/projected/2d3235a9-ace9-49c7-ae0a-5567ccacc120-kube-api-access-gdj54\") pod \"payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:33.666982 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.666917 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 00:59:33.725284 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:33.725255 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:34.114385 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.114356 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg"] Oct 31 00:59:34.115540 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:59:34.115519 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3235a9_ace9_49c7_ae0a_5567ccacc120.slice/crio-5d6af3458db9a9e55f7d604310d5319e574b7d0e37c6f186aada66b686577c33 WatchSource:0}: Error finding container 5d6af3458db9a9e55f7d604310d5319e574b7d0e37c6f186aada66b686577c33: Status 404 returned error can't find the container with id 5d6af3458db9a9e55f7d604310d5319e574b7d0e37c6f186aada66b686577c33 Oct 31 00:59:34.221616 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.221589 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf"] Oct 31 00:59:34.225744 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.225725 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.281446 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.281409 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf"] Oct 31 00:59:34.288050 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.288016 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wdq\" (UniqueName: \"kubernetes.io/projected/74216060-038e-4445-b056-91e9283a8af8-kube-api-access-d2wdq\") pod \"payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.288232 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.288091 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74216060-038e-4445-b056-91e9283a8af8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.389726 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.389639 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wdq\" (UniqueName: \"kubernetes.io/projected/74216060-038e-4445-b056-91e9283a8af8-kube-api-access-d2wdq\") pod \"payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.389726 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.389717 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74216060-038e-4445-b056-91e9283a8af8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.390665 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.390639 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74216060-038e-4445-b056-91e9283a8af8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.448685 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.448655 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wdq\" (UniqueName: \"kubernetes.io/projected/74216060-038e-4445-b056-91e9283a8af8-kube-api-access-d2wdq\") pod \"payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.539605 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.539574 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:34.754460 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.754411 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" event={"ID":"2d3235a9-ace9-49c7-ae0a-5567ccacc120","Type":"ContainerStarted","Data":"6f13639916a5e1b3891548cc4f99291d315dcc5281496d325b49dbcab96bfc71"} Oct 31 00:59:34.754649 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.754481 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" event={"ID":"2d3235a9-ace9-49c7-ae0a-5567ccacc120","Type":"ContainerStarted","Data":"5d6af3458db9a9e55f7d604310d5319e574b7d0e37c6f186aada66b686577c33"} Oct 31 00:59:34.754649 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.754512 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:34.872088 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.872025 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" podStartSLOduration=1.8720080110000001 podStartE2EDuration="1.872008011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:34.869158853 +0000 UTC m=+1867.071409286" watchObservedRunningTime="2026-10-31 00:59:34.872008011 +0000 UTC m=+1867.074258447" Oct 31 00:59:34.920674 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:34.920642 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf"] Oct 31 00:59:34.921223 ip-10-0-143-105 kubenswrapper[5890]: W1031 00:59:34.921198 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74216060_038e_4445_b056_91e9283a8af8.slice/crio-d7a0553b6095a08acf0ceb163aa625e9ffb782b13b30aeebc9fd478aa2a11694 WatchSource:0}: Error finding container d7a0553b6095a08acf0ceb163aa625e9ffb782b13b30aeebc9fd478aa2a11694: Status 404 returned error can't find the container with id d7a0553b6095a08acf0ceb163aa625e9ffb782b13b30aeebc9fd478aa2a11694 Oct 31 00:59:35.679904 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:35.679871 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 00:59:35.766138 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:35.766096 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" event={"ID":"74216060-038e-4445-b056-91e9283a8af8","Type":"ContainerStarted","Data":"23cf2129db7900c6142c7a1bb72329444acee863b821f2260ff3eb269d28a58c"} Oct 31 00:59:35.766430 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:35.766399 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" event={"ID":"74216060-038e-4445-b056-91e9283a8af8","Type":"ContainerStarted","Data":"d7a0553b6095a08acf0ceb163aa625e9ffb782b13b30aeebc9fd478aa2a11694"} Oct 31 00:59:35.766610 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:35.766588 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:37.019369 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:37.019328 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 00:59:45.769305 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:45.769273 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 00:59:45.883011 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:45.882958 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" podStartSLOduration=11.882944437999999 podStartE2EDuration="11.882944438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:35.90608954 +0000 UTC m=+1868.108339974" watchObservedRunningTime="2026-10-31 00:59:45.882944438 +0000 UTC m=+1878.085194901" Oct 31 00:59:47.783764 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:47.783734 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 00:59:49.198961 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.198925 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5"] Oct 31 00:59:49.199440 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.199171 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" podUID="5d4e2aba-5200-421c-80ba-d7e20a7ef582" containerName="payload-processing" containerID="cri-o://0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1" gracePeriod=30 Oct 31 00:59:49.482637 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.482610 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:59:49.592379 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.592346 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4e2aba-5200-421c-80ba-d7e20a7ef582-kube-api-access-b577d\" (UniqueName: \"kubernetes.io/projected/5d4e2aba-5200-421c-80ba-d7e20a7ef582-kube-api-access-b577d\") pod \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " Oct 31 00:59:49.592577 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.592559 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4e2aba-5200-421c-80ba-d7e20a7ef582-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e2aba-5200-421c-80ba-d7e20a7ef582-plugins-config-volume\") pod \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\" (UID: \"5d4e2aba-5200-421c-80ba-d7e20a7ef582\") " Oct 31 00:59:49.592937 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.592913 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4e2aba-5200-421c-80ba-d7e20a7ef582-plugins-config-volume" pod "5d4e2aba-5200-421c-80ba-d7e20a7ef582" (UID: "5d4e2aba-5200-421c-80ba-d7e20a7ef582"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:49.594543 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.594506 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4e2aba-5200-421c-80ba-d7e20a7ef582-kube-api-access-b577d" pod "5d4e2aba-5200-421c-80ba-d7e20a7ef582" (UID: "5d4e2aba-5200-421c-80ba-d7e20a7ef582"). InnerVolumeSpecName "kube-api-access-b577d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:49.720967 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.698887 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e2aba-5200-421c-80ba-d7e20a7ef582-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:49.721425 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.721395 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b577d\" (UniqueName: \"kubernetes.io/projected/5d4e2aba-5200-421c-80ba-d7e20a7ef582-kube-api-access-b577d\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:49.868852 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.868814 5890 generic.go:356] "Generic (PLEG): container finished" podID="5d4e2aba-5200-421c-80ba-d7e20a7ef582" containerID="0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1" exitCode=0 Oct 31 00:59:49.869019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.868870 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" Oct 31 00:59:49.869019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.868871 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" event={"ID":"5d4e2aba-5200-421c-80ba-d7e20a7ef582","Type":"ContainerDied","Data":"0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1"} Oct 31 00:59:49.869019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.868925 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5" event={"ID":"5d4e2aba-5200-421c-80ba-d7e20a7ef582","Type":"ContainerDied","Data":"6112e2df5f1e5129fbb420e36630398ac819170a37ca4968eda61bf6796442f6"} Oct 31 00:59:49.869019 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.868951 5890 scope.go:122] "RemoveContainer" containerID="0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1" Oct 31 00:59:49.879793 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.879759 5890 scope.go:122] "RemoveContainer" containerID="0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1" Oct 31 00:59:49.880094 ip-10-0-143-105 kubenswrapper[5890]: E1031 00:59:49.880075 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1\": container with ID starting with 0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1 not found: ID does not exist" containerID="0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1" Oct 31 00:59:49.880170 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:49.880107 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1"} err="failed to get container status \"0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1\": rpc error: code = NotFound desc = could not find container \"0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1\": container with ID starting with 0095249b9c60389598fc8b2ea26ee801f624db80a795a79b4e98bf53efffddb1 not found: ID does not exist" Oct 31 00:59:50.029109 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:50.029033 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5"] Oct 31 00:59:50.032034 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:50.032007 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66180-85458794b8-hw7f5"] Oct 31 00:59:51.933652 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:51.933618 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4e2aba-5200-421c-80ba-d7e20a7ef582" path="/var/lib/kubelet/pods/5d4e2aba-5200-421c-80ba-d7e20a7ef582/volumes" Oct 31 00:59:59.990598 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:59.990566 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x"] Oct 31 00:59:59.991161 ip-10-0-143-105 kubenswrapper[5890]: I1031 00:59:59.990794 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" containerID="cri-o://bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51" gracePeriod=30 Oct 31 01:00:00.178669 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.178614 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld"] Oct 31 01:00:00.179424 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.179399 5890 cpu_manager.go:414] "containerMap: removing container" podUID="5d4e2aba-5200-421c-80ba-d7e20a7ef582" containerName="payload-processing" Oct 31 01:00:00.179424 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.179421 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4e2aba-5200-421c-80ba-d7e20a7ef582" containerName="payload-processing" Oct 31 01:00:00.179580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.179533 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4e2aba-5200-421c-80ba-d7e20a7ef582" containerName="payload-processing" Oct 31 01:00:00.184305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.184284 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.194377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.194342 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf"] Oct 31 01:00:00.198562 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.198542 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.199287 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.199239 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cb9b35-825a-4742-90bc-54a581336b7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.199443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.199421 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/a9cb9b35-825a-4742-90bc-54a581336b7b-kube-api-access-57szj\") pod \"maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.207578 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.207548 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9"] Oct 31 01:00:00.212578 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.212558 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.243868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.243724 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:00:00.251577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.251545 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld"] Oct 31 01:00:00.273809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.272717 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf"] Oct 31 01:00:00.341805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.336894 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9"] Oct 31 01:00:00.341805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.339908 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23483b-31bf-4a77-90a7-68f93687dde1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.341805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.339990 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cb9b35-825a-4742-90bc-54a581336b7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.341805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.340067 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmh92\" (UniqueName: \"kubernetes.io/projected/844cbb90-9a30-4ede-9f66-38a8178b4811-kube-api-access-hmh92\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.341805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.340097 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8q8\" (UniqueName: \"kubernetes.io/projected/8d23483b-31bf-4a77-90a7-68f93687dde1-kube-api-access-5z8q8\") pod \"maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.341805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.340170 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844cbb90-9a30-4ede-9f66-38a8178b4811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.341805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.340256 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/a9cb9b35-825a-4742-90bc-54a581336b7b-kube-api-access-57szj\") pod \"maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.352754 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.343619 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cb9b35-825a-4742-90bc-54a581336b7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.407498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.407464 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/a9cb9b35-825a-4742-90bc-54a581336b7b-kube-api-access-57szj\") pod \"maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.442107 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.442073 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmh92\" (UniqueName: \"kubernetes.io/projected/844cbb90-9a30-4ede-9f66-38a8178b4811-kube-api-access-hmh92\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.442107 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.442110 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8q8\" (UniqueName: \"kubernetes.io/projected/8d23483b-31bf-4a77-90a7-68f93687dde1-kube-api-access-5z8q8\") pod \"maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.442317 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.442169 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844cbb90-9a30-4ede-9f66-38a8178b4811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.442422 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.442400 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23483b-31bf-4a77-90a7-68f93687dde1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.443015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.442996 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23483b-31bf-4a77-90a7-68f93687dde1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.443145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.443121 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844cbb90-9a30-4ede-9f66-38a8178b4811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.568694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.567671 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmh92\" (UniqueName: \"kubernetes.io/projected/844cbb90-9a30-4ede-9f66-38a8178b4811-kube-api-access-hmh92\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.568694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.568357 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8q8\" (UniqueName: \"kubernetes.io/projected/8d23483b-31bf-4a77-90a7-68f93687dde1-kube-api-access-5z8q8\") pod \"maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.570097 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.570070 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:00.582082 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.580834 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:00.584228 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.584198 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:00.977110 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:00.977075 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld"] Oct 31 01:00:00.983228 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:00:00.983200 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cb9b35_825a_4742_90bc_54a581336b7b.slice/crio-8ea37657ffccd07c0cd5c8c71a44c15185c6310b0a4262ed787c6cf2c2bb177b WatchSource:0}: Error finding container 8ea37657ffccd07c0cd5c8c71a44c15185c6310b0a4262ed787c6cf2c2bb177b: Status 404 returned error can't find the container with id 8ea37657ffccd07c0cd5c8c71a44c15185c6310b0a4262ed787c6cf2c2bb177b Oct 31 01:00:01.025979 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.025883 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9"] Oct 31 01:00:01.033190 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:00:01.033152 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d23483b_31bf_4a77_90a7_68f93687dde1.slice/crio-a8e2d38d44889851e6236f4e2dbfa10a0d2d28f096dc80e32be68c7f09e49103 WatchSource:0}: Error finding container a8e2d38d44889851e6236f4e2dbfa10a0d2d28f096dc80e32be68c7f09e49103: Status 404 returned error can't find the container with id a8e2d38d44889851e6236f4e2dbfa10a0d2d28f096dc80e32be68c7f09e49103 Oct 31 01:00:01.036023 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:00:01.036001 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844cbb90_9a30_4ede_9f66_38a8178b4811.slice/crio-1176f752b237307b68338f9ebc172e09d2e257276ad716cd1e238870f3d90647 WatchSource:0}: Error finding container 1176f752b237307b68338f9ebc172e09d2e257276ad716cd1e238870f3d90647: Status 404 returned error can't find the container with id 1176f752b237307b68338f9ebc172e09d2e257276ad716cd1e238870f3d90647 Oct 31 01:00:01.037867 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.037728 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf"] Oct 31 01:00:01.970123 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.969937 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" event={"ID":"8d23483b-31bf-4a77-90a7-68f93687dde1","Type":"ContainerStarted","Data":"d31ed2d1b21c2ad4c72339e650075b26ac41c6349923a67fa0910ab5e171428b"} Oct 31 01:00:01.970123 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.970003 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" event={"ID":"8d23483b-31bf-4a77-90a7-68f93687dde1","Type":"ContainerStarted","Data":"a8e2d38d44889851e6236f4e2dbfa10a0d2d28f096dc80e32be68c7f09e49103"} Oct 31 01:00:01.972310 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.972283 5890 generic.go:356] "Generic (PLEG): container finished" podID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerID="470a328c6c35ce7f398f254597594a148e44d7df7a5096527f4f63719a82eb92" exitCode=60 Oct 31 01:00:01.972440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.972366 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" event={"ID":"a9cb9b35-825a-4742-90bc-54a581336b7b","Type":"ContainerDied","Data":"470a328c6c35ce7f398f254597594a148e44d7df7a5096527f4f63719a82eb92"} Oct 31 01:00:01.972440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.972428 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" event={"ID":"a9cb9b35-825a-4742-90bc-54a581336b7b","Type":"ContainerStarted","Data":"8ea37657ffccd07c0cd5c8c71a44c15185c6310b0a4262ed787c6cf2c2bb177b"} Oct 31 01:00:01.972637 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.972622 5890 scope.go:122] "RemoveContainer" containerID="470a328c6c35ce7f398f254597594a148e44d7df7a5096527f4f63719a82eb92" Oct 31 01:00:01.974417 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.974327 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" event={"ID":"844cbb90-9a30-4ede-9f66-38a8178b4811","Type":"ContainerStarted","Data":"d64707c9b9e5993646d5db29fbe1682c7eb3af8e9a015d53d490ba92b0cabfbc"} Oct 31 01:00:01.974417 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:01.974377 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" event={"ID":"844cbb90-9a30-4ede-9f66-38a8178b4811","Type":"ContainerStarted","Data":"1176f752b237307b68338f9ebc172e09d2e257276ad716cd1e238870f3d90647"} Oct 31 01:00:02.083472 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.083416 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" podStartSLOduration=2.083399941 podStartE2EDuration="2.083399941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.079927385 +0000 UTC m=+1894.282177818" watchObservedRunningTime="2026-10-31 01:00:02.083399941 +0000 UTC m=+1894.285650374" Oct 31 01:00:02.302987 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.302930 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" podStartSLOduration=2.302915181 podStartE2EDuration="2.302915181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.302207127 +0000 UTC m=+1894.504457561" watchObservedRunningTime="2026-10-31 01:00:02.302915181 +0000 UTC m=+1894.505165614" Oct 31 01:00:02.984972 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.984937 5890 generic.go:356] "Generic (PLEG): container finished" podID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerID="d31ed2d1b21c2ad4c72339e650075b26ac41c6349923a67fa0910ab5e171428b" exitCode=60 Oct 31 01:00:02.985172 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.985028 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" event={"ID":"8d23483b-31bf-4a77-90a7-68f93687dde1","Type":"ContainerDied","Data":"d31ed2d1b21c2ad4c72339e650075b26ac41c6349923a67fa0910ab5e171428b"} Oct 31 01:00:02.985345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.985327 5890 scope.go:122] "RemoveContainer" containerID="d31ed2d1b21c2ad4c72339e650075b26ac41c6349923a67fa0910ab5e171428b" Oct 31 01:00:02.987476 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.987438 5890 generic.go:356] "Generic (PLEG): container finished" podID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerID="22d1c15a831d4a2495cf7c392f968c36b3722989b30d68fc0b7ede0fca6cd694" exitCode=60 Oct 31 01:00:02.987593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.987496 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" event={"ID":"a9cb9b35-825a-4742-90bc-54a581336b7b","Type":"ContainerDied","Data":"22d1c15a831d4a2495cf7c392f968c36b3722989b30d68fc0b7ede0fca6cd694"} Oct 31 01:00:02.987593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.987554 5890 scope.go:122] "RemoveContainer" containerID="470a328c6c35ce7f398f254597594a148e44d7df7a5096527f4f63719a82eb92" Oct 31 01:00:02.987834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.987815 5890 scope.go:122] "RemoveContainer" containerID="22d1c15a831d4a2495cf7c392f968c36b3722989b30d68fc0b7ede0fca6cd694" Oct 31 01:00:02.988867 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:02.988838 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld_odh-ai-gateway-infra(a9cb9b35-825a-4742-90bc-54a581336b7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" Oct 31 01:00:02.990772 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.990750 5890 generic.go:356] "Generic (PLEG): container finished" podID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerID="d64707c9b9e5993646d5db29fbe1682c7eb3af8e9a015d53d490ba92b0cabfbc" exitCode=60 Oct 31 01:00:02.990990 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.990930 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" event={"ID":"844cbb90-9a30-4ede-9f66-38a8178b4811","Type":"ContainerDied","Data":"d64707c9b9e5993646d5db29fbe1682c7eb3af8e9a015d53d490ba92b0cabfbc"} Oct 31 01:00:02.991176 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:02.991151 5890 scope.go:122] "RemoveContainer" containerID="d64707c9b9e5993646d5db29fbe1682c7eb3af8e9a015d53d490ba92b0cabfbc" Oct 31 01:00:04.026031 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:04.025928 5890 generic.go:356] "Generic (PLEG): container finished" podID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerID="40e1bb76032c9f9ef77acd863b777b51b861932b5c952572ecd745b730c775b3" exitCode=60 Oct 31 01:00:04.028470 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:04.028402 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" event={"ID":"8d23483b-31bf-4a77-90a7-68f93687dde1","Type":"ContainerDied","Data":"40e1bb76032c9f9ef77acd863b777b51b861932b5c952572ecd745b730c775b3"} Oct 31 01:00:04.038904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:04.029824 5890 scope.go:122] "RemoveContainer" containerID="40e1bb76032c9f9ef77acd863b777b51b861932b5c952572ecd745b730c775b3" Oct 31 01:00:04.038904 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:04.032758 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9_odh-ai-gateway-infra(8d23483b-31bf-4a77-90a7-68f93687dde1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" Oct 31 01:00:04.038904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:04.033033 5890 scope.go:122] "RemoveContainer" containerID="d31ed2d1b21c2ad4c72339e650075b26ac41c6349923a67fa0910ab5e171428b" Oct 31 01:00:04.038904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:04.036457 5890 scope.go:122] "RemoveContainer" containerID="22d1c15a831d4a2495cf7c392f968c36b3722989b30d68fc0b7ede0fca6cd694" Oct 31 01:00:04.038904 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:04.036707 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld_odh-ai-gateway-infra(a9cb9b35-825a-4742-90bc-54a581336b7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" Oct 31 01:00:04.038904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:04.037420 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" event={"ID":"844cbb90-9a30-4ede-9f66-38a8178b4811","Type":"ContainerStarted","Data":"8167a8e7efbebbc708d58446df25d2f7c45f6b6ef20fc014627a9fedd0c8883b"} Oct 31 01:00:05.047207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.047182 5890 scope.go:122] "RemoveContainer" containerID="40e1bb76032c9f9ef77acd863b777b51b861932b5c952572ecd745b730c775b3" Oct 31 01:00:05.048001 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:05.047975 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9_odh-ai-gateway-infra(8d23483b-31bf-4a77-90a7-68f93687dde1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" Oct 31 01:00:05.049313 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.049284 5890 generic.go:356] "Generic (PLEG): container finished" podID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerID="8167a8e7efbebbc708d58446df25d2f7c45f6b6ef20fc014627a9fedd0c8883b" exitCode=60 Oct 31 01:00:05.049416 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.049354 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" event={"ID":"844cbb90-9a30-4ede-9f66-38a8178b4811","Type":"ContainerDied","Data":"8167a8e7efbebbc708d58446df25d2f7c45f6b6ef20fc014627a9fedd0c8883b"} Oct 31 01:00:05.049416 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.049413 5890 scope.go:122] "RemoveContainer" containerID="d64707c9b9e5993646d5db29fbe1682c7eb3af8e9a015d53d490ba92b0cabfbc" Oct 31 01:00:05.049669 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.049646 5890 scope.go:122] "RemoveContainer" containerID="8167a8e7efbebbc708d58446df25d2f7c45f6b6ef20fc014627a9fedd0c8883b" Oct 31 01:00:05.050052 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:05.050028 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf_odh-ai-gateway-infra(844cbb90-9a30-4ede-9f66-38a8178b4811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" Oct 31 01:00:05.272877 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.272772 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 01:00:05.353425 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353390 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-socket\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.353615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353445 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-credential-socket\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.353615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353490 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-data\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.353615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353514 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39409f96-13f4-433b-98c8-e3c680e08f45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39409f96-13f4-433b-98c8-e3c680e08f45-istio-podinfo\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.353615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353583 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39409f96-13f4-433b-98c8-e3c680e08f45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39409f96-13f4-433b-98c8-e3c680e08f45-istiod-ca-cert\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.353852 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353678 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-envoy\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.353852 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353694 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-socket" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:05.353852 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353708 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-kube-api-access-bt6m2\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-kube-api-access-bt6m2\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.353852 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353822 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-certs\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.354056 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353871 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-credential-socket" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:05.354056 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353882 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-data" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:05.354056 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.353919 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-istio-token\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-istio-token\") pod \"39409f96-13f4-433b-98c8-e3c680e08f45\" (UID: \"39409f96-13f4-433b-98c8-e3c680e08f45\") " Oct 31 01:00:05.354191 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.354155 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-certs" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:05.354249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.354215 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39409f96-13f4-433b-98c8-e3c680e08f45-istiod-ca-cert" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:05.354598 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.354572 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.354598 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.354601 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.354803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.354619 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.354803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.354633 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39409f96-13f4-433b-98c8-e3c680e08f45-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.354803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.354648 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.356119 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.356047 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-kube-api-access-bt6m2" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "kube-api-access-bt6m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:05.356421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.356399 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39409f96-13f4-433b-98c8-e3c680e08f45-istio-podinfo" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:05.356677 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.356653 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-istio-token" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:05.356917 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.356894 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-envoy" pod "39409f96-13f4-433b-98c8-e3c680e08f45" (UID: "39409f96-13f4-433b-98c8-e3c680e08f45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:05.455893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.455861 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39409f96-13f4-433b-98c8-e3c680e08f45-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.455893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.455894 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6m2\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-kube-api-access-bt6m2\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.456087 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.455910 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39409f96-13f4-433b-98c8-e3c680e08f45-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:05.456087 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:05.455923 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39409f96-13f4-433b-98c8-e3c680e08f45-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:06.061186 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.061155 5890 scope.go:122] "RemoveContainer" containerID="8167a8e7efbebbc708d58446df25d2f7c45f6b6ef20fc014627a9fedd0c8883b" Oct 31 01:00:06.061652 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:06.061513 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf_odh-ai-gateway-infra(844cbb90-9a30-4ede-9f66-38a8178b4811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" Oct 31 01:00:06.062675 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.062645 5890 generic.go:356] "Generic (PLEG): container finished" podID="39409f96-13f4-433b-98c8-e3c680e08f45" containerID="bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51" exitCode=0 Oct 31 01:00:06.062842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.062693 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" Oct 31 01:00:06.062842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.062701 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" event={"ID":"39409f96-13f4-433b-98c8-e3c680e08f45","Type":"ContainerDied","Data":"bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51"} Oct 31 01:00:06.062842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.062826 5890 scope.go:122] "RemoveContainer" containerID="bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51" Oct 31 01:00:06.062998 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.062891 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x" event={"ID":"39409f96-13f4-433b-98c8-e3c680e08f45","Type":"ContainerDied","Data":"e30bbf9a8b1669b9fe7400543b63779f9920a2b843719227085f6cae05d93a5d"} Oct 31 01:00:06.074834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.074811 5890 scope.go:122] "RemoveContainer" containerID="bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51" Oct 31 01:00:06.075137 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:06.075114 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51\": container with ID starting with bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51 not found: ID does not exist" containerID="bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51" Oct 31 01:00:06.075238 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.075148 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51"} err="failed to get container status \"bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51\": rpc error: code = NotFound desc = could not find container \"bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51\": container with ID starting with bb333efe959b3c4a7d42307a26ba6f6e1fbc86c5d7f7e2ff7d8b650a788d7e51 not found: ID does not exist" Oct 31 01:00:06.282607 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.282520 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x"] Oct 31 01:00:06.327401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:06.327368 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hdx4x"] Oct 31 01:00:07.033383 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:07.033347 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:00:07.935196 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:07.935166 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" path="/var/lib/kubelet/pods/39409f96-13f4-433b-98c8-e3c680e08f45/volumes" Oct 31 01:00:13.926131 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:13.926057 5890 scope.go:122] "RemoveContainer" containerID="40e1bb76032c9f9ef77acd863b777b51b861932b5c952572ecd745b730c775b3" Oct 31 01:00:15.135822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.135700 5890 generic.go:356] "Generic (PLEG): container finished" podID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerID="96b2676e5290bd9349ddd571a3522b628007d46c52dec90a2c1e1eea8492451d" exitCode=60 Oct 31 01:00:15.136244 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.135777 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" event={"ID":"8d23483b-31bf-4a77-90a7-68f93687dde1","Type":"ContainerDied","Data":"96b2676e5290bd9349ddd571a3522b628007d46c52dec90a2c1e1eea8492451d"} Oct 31 01:00:15.136244 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.135888 5890 scope.go:122] "RemoveContainer" containerID="40e1bb76032c9f9ef77acd863b777b51b861932b5c952572ecd745b730c775b3" Oct 31 01:00:15.136370 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.136273 5890 scope.go:122] "RemoveContainer" containerID="96b2676e5290bd9349ddd571a3522b628007d46c52dec90a2c1e1eea8492451d" Oct 31 01:00:15.136643 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:15.136618 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9_odh-ai-gateway-infra(8d23483b-31bf-4a77-90a7-68f93687dde1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" Oct 31 01:00:15.532244 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.532214 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp"] Oct 31 01:00:15.532958 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.532941 5890 cpu_manager.go:414] "containerMap: removing container" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" Oct 31 01:00:15.533017 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.532959 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" Oct 31 01:00:15.533076 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.533066 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="39409f96-13f4-433b-98c8-e3c680e08f45" containerName="istio-proxy" Oct 31 01:00:15.537982 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.537966 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.582534 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.582504 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nz7p\" (UniqueName: \"kubernetes.io/projected/3bf6f33a-7b69-40d0-add0-2320348ca472-kube-api-access-8nz7p\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.582693 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.582568 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-metrics-certs\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.582739 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.582688 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.582827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.582811 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6f33a-7b69-40d0-add0-2320348ca472-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.588239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.588218 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8c08e7-serving-cert\"" Oct 31 01:00:15.631228 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.631197 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp"] Oct 31 01:00:15.691634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.691593 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.691824 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.691742 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6f33a-7b69-40d0-add0-2320348ca472-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.691908 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.691880 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nz7p\" (UniqueName: \"kubernetes.io/projected/3bf6f33a-7b69-40d0-add0-2320348ca472-kube-api-access-8nz7p\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.691971 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.691944 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-metrics-certs\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.693124 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.693082 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6f33a-7b69-40d0-add0-2320348ca472-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.693253 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:15.693189 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-serving-cert: secret "maas-api-e2e-praxis-a-8c08e7-serving-cert" not found Oct 31 01:00:15.693311 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:15.693253 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls podName:3bf6f33a-7b69-40d0-add0-2320348ca472 nodeName:}" failed. No retries permitted until 2026-10-31 01:00:16.193235288 +0000 UTC m=+1908.395485713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls") pod "maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" (UID: "3bf6f33a-7b69-40d0-add0-2320348ca472") : secret "maas-api-e2e-praxis-a-8c08e7-serving-cert" not found Oct 31 01:00:15.695214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.695190 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-metrics-certs\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.751948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.751920 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nz7p\" (UniqueName: \"kubernetes.io/projected/3bf6f33a-7b69-40d0-add0-2320348ca472-kube-api-access-8nz7p\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:15.935750 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.935719 5890 scope.go:122] "RemoveContainer" containerID="22d1c15a831d4a2495cf7c392f968c36b3722989b30d68fc0b7ede0fca6cd694" Oct 31 01:00:15.935937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:15.935888 5890 scope.go:122] "RemoveContainer" containerID="8167a8e7efbebbc708d58446df25d2f7c45f6b6ef20fc014627a9fedd0c8883b" Oct 31 01:00:16.199158 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.199075 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:16.201476 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.201445 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:16.268559 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.268526 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9"] Oct 31 01:00:16.452696 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.452345 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:16.490226 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.490199 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:16.604046 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.604010 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d23483b-31bf-4a77-90a7-68f93687dde1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23483b-31bf-4a77-90a7-68f93687dde1-maas-api-service-ca\") pod \"8d23483b-31bf-4a77-90a7-68f93687dde1\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " Oct 31 01:00:16.604307 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.604282 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d23483b-31bf-4a77-90a7-68f93687dde1-kube-api-access-5z8q8\" (UniqueName: \"kubernetes.io/projected/8d23483b-31bf-4a77-90a7-68f93687dde1-kube-api-access-5z8q8\") pod \"8d23483b-31bf-4a77-90a7-68f93687dde1\" (UID: \"8d23483b-31bf-4a77-90a7-68f93687dde1\") " Oct 31 01:00:16.604495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.604468 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d23483b-31bf-4a77-90a7-68f93687dde1-maas-api-service-ca" pod "8d23483b-31bf-4a77-90a7-68f93687dde1" (UID: "8d23483b-31bf-4a77-90a7-68f93687dde1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:16.604834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.604815 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d23483b-31bf-4a77-90a7-68f93687dde1-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:16.606550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.606527 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d23483b-31bf-4a77-90a7-68f93687dde1-kube-api-access-5z8q8" pod "8d23483b-31bf-4a77-90a7-68f93687dde1" (UID: "8d23483b-31bf-4a77-90a7-68f93687dde1"). InnerVolumeSpecName "kube-api-access-5z8q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:16.706532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.706505 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8q8\" (UniqueName: \"kubernetes.io/projected/8d23483b-31bf-4a77-90a7-68f93687dde1-kube-api-access-5z8q8\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:16.843399 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:16.843353 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp"] Oct 31 01:00:16.844288 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:00:16.844257 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf6f33a_7b69_40d0_add0_2320348ca472.slice/crio-731201cadfe6c893da02047c89851d4d0fa00320ec675390902a7a1aef942c04 WatchSource:0}: Error finding container 731201cadfe6c893da02047c89851d4d0fa00320ec675390902a7a1aef942c04: Status 404 returned error can't find the container with id 731201cadfe6c893da02047c89851d4d0fa00320ec675390902a7a1aef942c04 Oct 31 01:00:17.155286 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.155257 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" Oct 31 01:00:17.155624 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.155253 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9" event={"ID":"8d23483b-31bf-4a77-90a7-68f93687dde1","Type":"ContainerDied","Data":"a8e2d38d44889851e6236f4e2dbfa10a0d2d28f096dc80e32be68c7f09e49103"} Oct 31 01:00:17.155624 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.155412 5890 scope.go:122] "RemoveContainer" containerID="96b2676e5290bd9349ddd571a3522b628007d46c52dec90a2c1e1eea8492451d" Oct 31 01:00:17.157429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.157406 5890 generic.go:356] "Generic (PLEG): container finished" podID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerID="04480850ebfc41d49801135087e9c7e3afd24ba44f93f4f555febde4fdda5e5b" exitCode=60 Oct 31 01:00:17.157535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.157457 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" event={"ID":"a9cb9b35-825a-4742-90bc-54a581336b7b","Type":"ContainerDied","Data":"04480850ebfc41d49801135087e9c7e3afd24ba44f93f4f555febde4fdda5e5b"} Oct 31 01:00:17.157980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.157955 5890 scope.go:122] "RemoveContainer" containerID="04480850ebfc41d49801135087e9c7e3afd24ba44f93f4f555febde4fdda5e5b" Oct 31 01:00:17.158312 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:17.158282 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld_odh-ai-gateway-infra(a9cb9b35-825a-4742-90bc-54a581336b7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" Oct 31 01:00:17.160335 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.160309 5890 generic.go:356] "Generic (PLEG): container finished" podID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerID="2df10740781694a3b53cea68dd181a9061d0e05a9af8d8ec9def8f22f0c77af9" exitCode=60 Oct 31 01:00:17.160448 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.160424 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" event={"ID":"844cbb90-9a30-4ede-9f66-38a8178b4811","Type":"ContainerDied","Data":"2df10740781694a3b53cea68dd181a9061d0e05a9af8d8ec9def8f22f0c77af9"} Oct 31 01:00:17.160871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.160848 5890 scope.go:122] "RemoveContainer" containerID="2df10740781694a3b53cea68dd181a9061d0e05a9af8d8ec9def8f22f0c77af9" Oct 31 01:00:17.161167 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:17.161144 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf_odh-ai-gateway-infra(844cbb90-9a30-4ede-9f66-38a8178b4811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" Oct 31 01:00:17.162184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.162158 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" event={"ID":"3bf6f33a-7b69-40d0-add0-2320348ca472","Type":"ContainerStarted","Data":"731201cadfe6c893da02047c89851d4d0fa00320ec675390902a7a1aef942c04"} Oct 31 01:00:17.167772 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.167750 5890 scope.go:122] "RemoveContainer" containerID="22d1c15a831d4a2495cf7c392f968c36b3722989b30d68fc0b7ede0fca6cd694" Oct 31 01:00:17.181101 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.181076 5890 scope.go:122] "RemoveContainer" containerID="8167a8e7efbebbc708d58446df25d2f7c45f6b6ef20fc014627a9fedd0c8883b" Oct 31 01:00:17.424445 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.424391 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9"] Oct 31 01:00:17.428731 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.428702 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eee7f1-29890140-88kk9"] Oct 31 01:00:17.935962 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:17.935932 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" path="/var/lib/kubelet/pods/8d23483b-31bf-4a77-90a7-68f93687dde1/volumes" Oct 31 01:00:18.192765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.192728 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" event={"ID":"3bf6f33a-7b69-40d0-add0-2320348ca472","Type":"ContainerStarted","Data":"7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15"} Oct 31 01:00:18.192942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.192861 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:18.321058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.320999 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" podStartSLOduration=2.343136769 podStartE2EDuration="3.3209851s" totalImagesPullingTime="977.848331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:18.318354973 +0000 UTC m=+1910.520605408" watchObservedRunningTime="2026-10-31 01:00:18.3209851 +0000 UTC m=+1910.523235532" Oct 31 01:00:18.369851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.369815 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf"] Oct 31 01:00:18.549661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.549595 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld"] Oct 31 01:00:18.576855 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.576821 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:18.640432 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.640394 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844cbb90-9a30-4ede-9f66-38a8178b4811-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844cbb90-9a30-4ede-9f66-38a8178b4811-maas-api-service-ca\") pod \"844cbb90-9a30-4ede-9f66-38a8178b4811\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " Oct 31 01:00:18.642488 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.640727 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844cbb90-9a30-4ede-9f66-38a8178b4811-kube-api-access-hmh92\" (UniqueName: \"kubernetes.io/projected/844cbb90-9a30-4ede-9f66-38a8178b4811-kube-api-access-hmh92\") pod \"844cbb90-9a30-4ede-9f66-38a8178b4811\" (UID: \"844cbb90-9a30-4ede-9f66-38a8178b4811\") " Oct 31 01:00:18.646082 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.645773 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844cbb90-9a30-4ede-9f66-38a8178b4811-kube-api-access-hmh92" pod "844cbb90-9a30-4ede-9f66-38a8178b4811" (UID: "844cbb90-9a30-4ede-9f66-38a8178b4811"). InnerVolumeSpecName "kube-api-access-hmh92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.647623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.646304 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844cbb90-9a30-4ede-9f66-38a8178b4811-maas-api-service-ca" pod "844cbb90-9a30-4ede-9f66-38a8178b4811" (UID: "844cbb90-9a30-4ede-9f66-38a8178b4811"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.746368 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.744564 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844cbb90-9a30-4ede-9f66-38a8178b4811-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.746368 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.744622 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmh92\" (UniqueName: \"kubernetes.io/projected/844cbb90-9a30-4ede-9f66-38a8178b4811-kube-api-access-hmh92\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.770668 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.770640 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:18.845324 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.845292 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cb9b35-825a-4742-90bc-54a581336b7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cb9b35-825a-4742-90bc-54a581336b7b-maas-api-service-ca\") pod \"a9cb9b35-825a-4742-90bc-54a581336b7b\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " Oct 31 01:00:18.845602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.845583 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cb9b35-825a-4742-90bc-54a581336b7b-kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/a9cb9b35-825a-4742-90bc-54a581336b7b-kube-api-access-57szj\") pod \"a9cb9b35-825a-4742-90bc-54a581336b7b\" (UID: \"a9cb9b35-825a-4742-90bc-54a581336b7b\") " Oct 31 01:00:18.845761 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.845739 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cb9b35-825a-4742-90bc-54a581336b7b-maas-api-service-ca" pod "a9cb9b35-825a-4742-90bc-54a581336b7b" (UID: "a9cb9b35-825a-4742-90bc-54a581336b7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.846092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.846075 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cb9b35-825a-4742-90bc-54a581336b7b-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.847719 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.847691 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cb9b35-825a-4742-90bc-54a581336b7b-kube-api-access-57szj" pod "a9cb9b35-825a-4742-90bc-54a581336b7b" (UID: "a9cb9b35-825a-4742-90bc-54a581336b7b"). InnerVolumeSpecName "kube-api-access-57szj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.947115 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:18.947063 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57szj\" (UniqueName: \"kubernetes.io/projected/a9cb9b35-825a-4742-90bc-54a581336b7b-kube-api-access-57szj\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.203521 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.203488 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" Oct 31 01:00:19.203718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.203514 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld" event={"ID":"a9cb9b35-825a-4742-90bc-54a581336b7b","Type":"ContainerDied","Data":"8ea37657ffccd07c0cd5c8c71a44c15185c6310b0a4262ed787c6cf2c2bb177b"} Oct 31 01:00:19.203718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.203591 5890 scope.go:122] "RemoveContainer" containerID="04480850ebfc41d49801135087e9c7e3afd24ba44f93f4f555febde4fdda5e5b" Oct 31 01:00:19.205653 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.205619 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" event={"ID":"844cbb90-9a30-4ede-9f66-38a8178b4811","Type":"ContainerDied","Data":"1176f752b237307b68338f9ebc172e09d2e257276ad716cd1e238870f3d90647"} Oct 31 01:00:19.205979 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.205637 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf" Oct 31 01:00:19.216599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.216576 5890 scope.go:122] "RemoveContainer" containerID="2df10740781694a3b53cea68dd181a9061d0e05a9af8d8ec9def8f22f0c77af9" Oct 31 01:00:19.365166 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.365127 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld"] Oct 31 01:00:19.368593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.368565 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b647c3-29890140-wj5ld"] Oct 31 01:00:19.474883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.473392 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf"] Oct 31 01:00:19.476756 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.476726 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890140-g9fgf"] Oct 31 01:00:19.933947 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.933917 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" path="/var/lib/kubelet/pods/844cbb90-9a30-4ede-9f66-38a8178b4811/volumes" Oct 31 01:00:19.934415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:19.934400 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" path="/var/lib/kubelet/pods/a9cb9b35-825a-4742-90bc-54a581336b7b/volumes" Oct 31 01:00:24.211560 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:24.211530 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:00:27.157459 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.157422 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v"] Oct 31 01:00:27.158338 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158318 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158340 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158371 5890 cpu_manager.go:414] "containerMap: removing container" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158379 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158394 5890 cpu_manager.go:414] "containerMap: removing container" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158401 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158415 5890 cpu_manager.go:414] "containerMap: removing container" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158422 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158432 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:00:27.158460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158439 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158476 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158484 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158514 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158521 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158686 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158702 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158712 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158723 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158733 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="844cbb90-9a30-4ede-9f66-38a8178b4811" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158744 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158757 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:00:27.158916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.158770 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:00:27.162699 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.162676 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.214161 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.214133 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eb41d3-serving-cert\"" Oct 31 01:00:27.265030 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.264996 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v"] Oct 31 01:00:27.360129 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.360097 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.360129 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.360135 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6285afa6-04d6-456a-a7f8-9fe220d34e5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.360346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.360285 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbl6\" (UniqueName: \"kubernetes.io/projected/6285afa6-04d6-456a-a7f8-9fe220d34e5b-kube-api-access-wqbl6\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.360384 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.360347 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.461283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.461245 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.461467 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.461398 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.461467 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.461422 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6285afa6-04d6-456a-a7f8-9fe220d34e5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.461599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.461497 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbl6\" (UniqueName: \"kubernetes.io/projected/6285afa6-04d6-456a-a7f8-9fe220d34e5b-kube-api-access-wqbl6\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.462135 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.462107 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6285afa6-04d6-456a-a7f8-9fe220d34e5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.463774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.463754 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.463894 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.463825 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.520183 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.520149 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbl6\" (UniqueName: \"kubernetes.io/projected/6285afa6-04d6-456a-a7f8-9fe220d34e5b-kube-api-access-wqbl6\") pod \"maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:27.777827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:27.777726 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:28.151115 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:28.151076 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v"] Oct 31 01:00:28.151877 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:00:28.151848 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6285afa6_04d6_456a_a7f8_9fe220d34e5b.slice/crio-12efddc342415bacf1a621eb6c86b8097d57fca58f7d360f0b5c1abe479072d1 WatchSource:0}: Error finding container 12efddc342415bacf1a621eb6c86b8097d57fca58f7d360f0b5c1abe479072d1: Status 404 returned error can't find the container with id 12efddc342415bacf1a621eb6c86b8097d57fca58f7d360f0b5c1abe479072d1 Oct 31 01:00:28.280503 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:28.280454 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" event={"ID":"6285afa6-04d6-456a-a7f8-9fe220d34e5b","Type":"ContainerStarted","Data":"12efddc342415bacf1a621eb6c86b8097d57fca58f7d360f0b5c1abe479072d1"} Oct 31 01:00:29.288904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:29.288770 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" event={"ID":"6285afa6-04d6-456a-a7f8-9fe220d34e5b","Type":"ContainerStarted","Data":"4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9"} Oct 31 01:00:29.288904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:29.288882 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:29.412605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:29.412552 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" podStartSLOduration=1.453836666 podStartE2EDuration="2.412538537s" totalImagesPullingTime="958.701871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:29.411956799 +0000 UTC m=+1921.614207236" watchObservedRunningTime="2026-10-31 01:00:29.412538537 +0000 UTC m=+1921.614788997" Oct 31 01:00:35.301422 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:35.301389 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:00:37.017774 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:00:37.017731 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:00:54.543282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:54.543250 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb_b6b7291a-105a-4cee-9d55-a5ae80e1e571/payload-processing/0.log" Oct 31 01:00:55.277462 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:55.277429 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf_74216060-038e-4445-b056-91e9283a8af8/payload-processing/0.log" Oct 31 01:00:55.499893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:00:55.499867 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf_74216060-038e-4445-b056-91e9283a8af8/payload-processing/0.log" Oct 31 01:01:07.018199 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:01:07.018155 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:01:09.134536 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.134499 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp"] Oct 31 01:01:09.135065 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.134797 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" podUID="195c7669-1d5e-413f-9827-d0ada1bf8e09" containerName="maas-api" containerID="cri-o://8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d" gracePeriod=30 Oct 31 01:01:09.432280 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.432254 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 01:01:09.482268 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.482232 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-maas-api-tls\") pod \"195c7669-1d5e-413f-9827-d0ada1bf8e09\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " Oct 31 01:01:09.482467 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.482302 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195c7669-1d5e-413f-9827-d0ada1bf8e09-kube-api-access-r6w4r\" (UniqueName: \"kubernetes.io/projected/195c7669-1d5e-413f-9827-d0ada1bf8e09-kube-api-access-r6w4r\") pod \"195c7669-1d5e-413f-9827-d0ada1bf8e09\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " Oct 31 01:01:09.482467 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.482345 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-metrics-certs\") pod \"195c7669-1d5e-413f-9827-d0ada1bf8e09\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " Oct 31 01:01:09.482467 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.482384 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195c7669-1d5e-413f-9827-d0ada1bf8e09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c7669-1d5e-413f-9827-d0ada1bf8e09-openshift-service-ca\") pod \"195c7669-1d5e-413f-9827-d0ada1bf8e09\" (UID: \"195c7669-1d5e-413f-9827-d0ada1bf8e09\") " Oct 31 01:01:09.483175 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.483131 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195c7669-1d5e-413f-9827-d0ada1bf8e09-openshift-service-ca" pod "195c7669-1d5e-413f-9827-d0ada1bf8e09" (UID: "195c7669-1d5e-413f-9827-d0ada1bf8e09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:09.485447 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.485406 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-maas-api-tls" pod "195c7669-1d5e-413f-9827-d0ada1bf8e09" (UID: "195c7669-1d5e-413f-9827-d0ada1bf8e09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:09.485561 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.485492 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-metrics-certs" pod "195c7669-1d5e-413f-9827-d0ada1bf8e09" (UID: "195c7669-1d5e-413f-9827-d0ada1bf8e09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:09.485777 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.485760 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195c7669-1d5e-413f-9827-d0ada1bf8e09-kube-api-access-r6w4r" pod "195c7669-1d5e-413f-9827-d0ada1bf8e09" (UID: "195c7669-1d5e-413f-9827-d0ada1bf8e09"). InnerVolumeSpecName "kube-api-access-r6w4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:09.584495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.584459 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:09.584736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.584714 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6w4r\" (UniqueName: \"kubernetes.io/projected/195c7669-1d5e-413f-9827-d0ada1bf8e09-kube-api-access-r6w4r\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:09.584884 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.584867 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195c7669-1d5e-413f-9827-d0ada1bf8e09-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:09.584992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.584978 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c7669-1d5e-413f-9827-d0ada1bf8e09-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:09.592372 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.592335 5890 generic.go:356] "Generic (PLEG): container finished" podID="195c7669-1d5e-413f-9827-d0ada1bf8e09" containerID="8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d" exitCode=0 Oct 31 01:01:09.592518 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.592498 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" event={"ID":"195c7669-1d5e-413f-9827-d0ada1bf8e09","Type":"ContainerDied","Data":"8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d"} Oct 31 01:01:09.592588 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.592554 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" event={"ID":"195c7669-1d5e-413f-9827-d0ada1bf8e09","Type":"ContainerDied","Data":"d53e982e221679020bd13bb3cb3e6548ed1cb115a9cce0ff2ab056b9b74dd2eb"} Oct 31 01:01:09.592588 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.592583 5890 scope.go:122] "RemoveContainer" containerID="8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d" Oct 31 01:01:09.592797 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.592771 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp" Oct 31 01:01:09.613630 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.613608 5890 scope.go:122] "RemoveContainer" containerID="8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d" Oct 31 01:01:09.614002 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:01:09.613976 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d\": container with ID starting with 8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d not found: ID does not exist" containerID="8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d" Oct 31 01:01:09.614085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.614015 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d"} err="failed to get container status \"8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d\": rpc error: code = NotFound desc = could not find container \"8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d\": container with ID starting with 8d0f88ba31b9d5e66a59c4b0f79056a108a96f81350261e9413c715f8791172d not found: ID does not exist" Oct 31 01:01:09.761390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.761354 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp"] Oct 31 01:01:09.766593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.766559 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f6979f-76f5b767b-l9wvp"] Oct 31 01:01:09.935238 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:09.935203 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195c7669-1d5e-413f-9827-d0ada1bf8e09" path="/var/lib/kubelet/pods/195c7669-1d5e-413f-9827-d0ada1bf8e09/volumes" Oct 31 01:01:34.459644 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.459605 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp"] Oct 31 01:01:34.460324 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460307 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:01:34.460371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460325 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cb9b35-825a-4742-90bc-54a581336b7b" containerName="cleanup" Oct 31 01:01:34.460371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460337 5890 cpu_manager.go:414] "containerMap: removing container" podUID="195c7669-1d5e-413f-9827-d0ada1bf8e09" containerName="maas-api" Oct 31 01:01:34.460371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460343 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195c7669-1d5e-413f-9827-d0ada1bf8e09" containerName="maas-api" Oct 31 01:01:34.460371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460355 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:01:34.460371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460361 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:01:34.460527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460467 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="195c7669-1d5e-413f-9827-d0ada1bf8e09" containerName="maas-api" Oct 31 01:01:34.460527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.460476 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d23483b-31bf-4a77-90a7-68f93687dde1" containerName="cleanup" Oct 31 01:01:34.465398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.465376 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516408 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istiod-ca-cert\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516505 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-socket\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516534 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-token\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516569 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-data\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516614 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-certs\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516675 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-podinfo\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516710 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4rw\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-kube-api-access-wd4rw\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516776 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-credential-socket\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.522810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.516863 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-envoy\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.589842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.550457 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9e4801-openshift-default-dockercfg-ttsck\"" Oct 31 01:01:34.591369 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.591315 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp"] Oct 31 01:01:34.618065 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618031 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-envoy\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618091 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istiod-ca-cert\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618158 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-socket\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618185 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-token\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618222 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-data\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618274 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-certs\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618347 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-podinfo\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618388 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4rw\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-kube-api-access-wd4rw\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618473 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-credential-socket\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618569 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-socket\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618801 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618710 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-data\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618847 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-certs\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.618925 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.618871 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-credential-socket\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.619634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.619608 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istiod-ca-cert\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.621991 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.621964 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-podinfo\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.623371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.623342 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-envoy\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.678010 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.677976 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-token\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.678162 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.678138 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4rw\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-kube-api-access-wd4rw\") pod \"e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:34.783930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:34.783845 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:35.215836 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:35.215803 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp"] Oct 31 01:01:35.218448 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:01:35.218406 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50379ab5_7dc7_4d22_9ea8_194e4be2aba5.slice/crio-c6d282032fd95e44885507fb2132335aa88cf69f3a0270a87c6b40859cb0bfcd WatchSource:0}: Error finding container c6d282032fd95e44885507fb2132335aa88cf69f3a0270a87c6b40859cb0bfcd: Status 404 returned error can't find the container with id c6d282032fd95e44885507fb2132335aa88cf69f3a0270a87c6b40859cb0bfcd Oct 31 01:01:35.221580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:35.221546 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:01:35.221660 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:35.221616 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:01:35.221660 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:35.221645 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:01:35.797875 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:35.797836 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" event={"ID":"50379ab5-7dc7-4d22-9ea8-194e4be2aba5","Type":"ContainerStarted","Data":"63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657"} Oct 31 01:01:35.798317 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:35.797897 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" event={"ID":"50379ab5-7dc7-4d22-9ea8-194e4be2aba5","Type":"ContainerStarted","Data":"c6d282032fd95e44885507fb2132335aa88cf69f3a0270a87c6b40859cb0bfcd"} Oct 31 01:01:35.917944 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:35.917837 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podStartSLOduration=1.917822443 podStartE2EDuration="1.917822443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:35.914454233 +0000 UTC m=+1988.116704667" watchObservedRunningTime="2026-10-31 01:01:35.917822443 +0000 UTC m=+1988.120072875" Oct 31 01:01:36.784951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:36.784915 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:36.790819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:36.790758 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:36.806439 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:36.806400 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:36.807249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:36.807227 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:01:37.019321 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:01:37.019282 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:01:38.180859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:38.180824 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x"] Oct 31 01:01:38.278602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:38.278562 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n"] Oct 31 01:01:43.181756 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.181713 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" podUID="0562d880-8f7f-4673-9125-273738a1a76f" containerName="payload-processing" containerID="cri-o://5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b" gracePeriod=25 Oct 31 01:01:43.278905 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.278862 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" podUID="a7145009-3452-4f30-b2e5-6cae7c650b8b" containerName="payload-pre-processing" containerID="cri-o://9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298" gracePeriod=25 Oct 31 01:01:43.456530 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.456503 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 01:01:43.485963 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.485923 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0562d880-8f7f-4673-9125-273738a1a76f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0562d880-8f7f-4673-9125-273738a1a76f-plugins-config-volume\") pod \"0562d880-8f7f-4673-9125-273738a1a76f\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " Oct 31 01:01:43.486309 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.486282 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0562d880-8f7f-4673-9125-273738a1a76f-kube-api-access-rwg2b\" (UniqueName: \"kubernetes.io/projected/0562d880-8f7f-4673-9125-273738a1a76f-kube-api-access-rwg2b\") pod \"0562d880-8f7f-4673-9125-273738a1a76f\" (UID: \"0562d880-8f7f-4673-9125-273738a1a76f\") " Oct 31 01:01:43.486520 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.486499 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0562d880-8f7f-4673-9125-273738a1a76f-plugins-config-volume" pod "0562d880-8f7f-4673-9125-273738a1a76f" (UID: "0562d880-8f7f-4673-9125-273738a1a76f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:43.486962 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.486933 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0562d880-8f7f-4673-9125-273738a1a76f-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:43.489773 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.489743 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0562d880-8f7f-4673-9125-273738a1a76f-kube-api-access-rwg2b" pod "0562d880-8f7f-4673-9125-273738a1a76f" (UID: "0562d880-8f7f-4673-9125-273738a1a76f"). InnerVolumeSpecName "kube-api-access-rwg2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:43.551522 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.551495 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 01:01:43.593598 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.593556 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7145009-3452-4f30-b2e5-6cae7c650b8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7145009-3452-4f30-b2e5-6cae7c650b8b-plugins-config-volume\") pod \"a7145009-3452-4f30-b2e5-6cae7c650b8b\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " Oct 31 01:01:43.593849 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.593820 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7145009-3452-4f30-b2e5-6cae7c650b8b-kube-api-access-kdpzx\" (UniqueName: \"kubernetes.io/projected/a7145009-3452-4f30-b2e5-6cae7c650b8b-kube-api-access-kdpzx\") pod \"a7145009-3452-4f30-b2e5-6cae7c650b8b\" (UID: \"a7145009-3452-4f30-b2e5-6cae7c650b8b\") " Oct 31 01:01:43.595722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.594830 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7145009-3452-4f30-b2e5-6cae7c650b8b-plugins-config-volume" pod "a7145009-3452-4f30-b2e5-6cae7c650b8b" (UID: "a7145009-3452-4f30-b2e5-6cae7c650b8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:43.595722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.595017 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7145009-3452-4f30-b2e5-6cae7c650b8b-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:43.595722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.595037 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwg2b\" (UniqueName: \"kubernetes.io/projected/0562d880-8f7f-4673-9125-273738a1a76f-kube-api-access-rwg2b\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:43.596818 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.596753 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7145009-3452-4f30-b2e5-6cae7c650b8b-kube-api-access-kdpzx" pod "a7145009-3452-4f30-b2e5-6cae7c650b8b" (UID: "a7145009-3452-4f30-b2e5-6cae7c650b8b"). InnerVolumeSpecName "kube-api-access-kdpzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:43.696259 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.696185 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpzx\" (UniqueName: \"kubernetes.io/projected/a7145009-3452-4f30-b2e5-6cae7c650b8b-kube-api-access-kdpzx\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:43.867881 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.867840 5890 generic.go:356] "Generic (PLEG): container finished" podID="0562d880-8f7f-4673-9125-273738a1a76f" containerID="5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b" exitCode=0 Oct 31 01:01:43.868073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.867899 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" Oct 31 01:01:43.868073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.867934 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" event={"ID":"0562d880-8f7f-4673-9125-273738a1a76f","Type":"ContainerDied","Data":"5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b"} Oct 31 01:01:43.868073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.867990 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x" event={"ID":"0562d880-8f7f-4673-9125-273738a1a76f","Type":"ContainerDied","Data":"3eae4dd0172f18a0f8c1a83d4cf0086811cdc7b6c922ca0f6d69b42565e2544e"} Oct 31 01:01:43.868073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.868021 5890 scope.go:122] "RemoveContainer" containerID="5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b" Oct 31 01:01:43.869385 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.869360 5890 generic.go:356] "Generic (PLEG): container finished" podID="a7145009-3452-4f30-b2e5-6cae7c650b8b" containerID="9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298" exitCode=0 Oct 31 01:01:43.869472 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.869405 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" Oct 31 01:01:43.869472 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.869416 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" event={"ID":"a7145009-3452-4f30-b2e5-6cae7c650b8b","Type":"ContainerDied","Data":"9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298"} Oct 31 01:01:43.869573 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.869472 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n" event={"ID":"a7145009-3452-4f30-b2e5-6cae7c650b8b","Type":"ContainerDied","Data":"3442deea443851e02350977f4bb30c8ad0dee9d044449938b10aab7d1a4035b5"} Oct 31 01:01:43.879125 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.879011 5890 scope.go:122] "RemoveContainer" containerID="5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b" Oct 31 01:01:43.879411 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:01:43.879387 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b\": container with ID starting with 5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b not found: ID does not exist" containerID="5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b" Oct 31 01:01:43.879486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.879421 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b"} err="failed to get container status \"5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b\": rpc error: code = NotFound desc = could not find container \"5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b\": container with ID starting with 5257f1895d4a20f641e1d8d885164807e1529e42033df329e8d190b6eb6efc2b not found: ID does not exist" Oct 31 01:01:43.879486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.879446 5890 scope.go:122] "RemoveContainer" containerID="9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298" Oct 31 01:01:43.892431 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.892410 5890 scope.go:122] "RemoveContainer" containerID="9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298" Oct 31 01:01:43.892769 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:01:43.892739 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298\": container with ID starting with 9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298 not found: ID does not exist" containerID="9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298" Oct 31 01:01:43.892872 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:43.892813 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298"} err="failed to get container status \"9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298\": rpc error: code = NotFound desc = could not find container \"9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298\": container with ID starting with 9012b201823fb137259ce5fe5fbcae1245ccef2862b0e0d1a79808929d439298 not found: ID does not exist" Oct 31 01:01:44.028768 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:44.028681 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x"] Oct 31 01:01:44.032289 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:44.032266 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb41d3-5544bdb68f-nbc6x"] Oct 31 01:01:44.138317 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:44.138280 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n"] Oct 31 01:01:44.143577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:44.143542 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb41d3-59c66658cf-9zl2n"] Oct 31 01:01:45.933534 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:45.933500 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0562d880-8f7f-4673-9125-273738a1a76f" path="/var/lib/kubelet/pods/0562d880-8f7f-4673-9125-273738a1a76f/volumes" Oct 31 01:01:45.933980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:45.933965 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7145009-3452-4f30-b2e5-6cae7c650b8b" path="/var/lib/kubelet/pods/a7145009-3452-4f30-b2e5-6cae7c650b8b/volumes" Oct 31 01:01:54.254229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.254192 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n"] Oct 31 01:01:54.255084 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.255051 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a7145009-3452-4f30-b2e5-6cae7c650b8b" containerName="payload-pre-processing" Oct 31 01:01:54.255206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.255084 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7145009-3452-4f30-b2e5-6cae7c650b8b" containerName="payload-pre-processing" Oct 31 01:01:54.255206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.255117 5890 cpu_manager.go:414] "containerMap: removing container" podUID="0562d880-8f7f-4673-9125-273738a1a76f" containerName="payload-processing" Oct 31 01:01:54.255206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.255126 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0562d880-8f7f-4673-9125-273738a1a76f" containerName="payload-processing" Oct 31 01:01:54.255371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.255287 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7145009-3452-4f30-b2e5-6cae7c650b8b" containerName="payload-pre-processing" Oct 31 01:01:54.255371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.255307 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="0562d880-8f7f-4673-9125-273738a1a76f" containerName="payload-processing" Oct 31 01:01:54.258835 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.258814 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.330639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.330587 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.330846 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.330729 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8nf7\" (UniqueName: \"kubernetes.io/projected/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-kube-api-access-w8nf7\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.331055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.331029 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-metrics-certs\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.331167 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.331145 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-openshift-service-ca\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.358861 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.358822 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-31ae40-serving-cert\"" Oct 31 01:01:54.364253 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.364223 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n"] Oct 31 01:01:54.432611 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.432567 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-openshift-service-ca\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.432844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.432624 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.432844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.432686 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8nf7\" (UniqueName: \"kubernetes.io/projected/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-kube-api-access-w8nf7\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.432844 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:01:54.432709 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-serving-cert: secret "maas-api-e2e-api-b-31ae40-serving-cert" not found Oct 31 01:01:54.432844 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:01:54.432766 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls podName:fdee4d51-fbf2-416f-ad9d-f57e651a4ff0 nodeName:}" failed. No retries permitted until 2026-10-31 01:01:54.932752059 +0000 UTC m=+2007.135002470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls") pod "maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" (UID: "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0") : secret "maas-api-e2e-api-b-31ae40-serving-cert" not found Oct 31 01:01:54.433094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.432929 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-metrics-certs\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.433314 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.433294 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-openshift-service-ca\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.439410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.439381 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-metrics-certs\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.497519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.497480 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8nf7\" (UniqueName: \"kubernetes.io/projected/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-kube-api-access-w8nf7\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.939198 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.939165 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:54.941636 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:54.941614 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls\") pod \"maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:55.173955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:55.173879 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:55.562743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:55.562714 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n"] Oct 31 01:01:55.563350 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:01:55.563315 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdee4d51_fbf2_416f_ad9d_f57e651a4ff0.slice/crio-d5b7e977af758a09273c1b94acdce28e1989f8a16d3dffbdadb53159af870a8f WatchSource:0}: Error finding container d5b7e977af758a09273c1b94acdce28e1989f8a16d3dffbdadb53159af870a8f: Status 404 returned error can't find the container with id d5b7e977af758a09273c1b94acdce28e1989f8a16d3dffbdadb53159af870a8f Oct 31 01:01:55.967258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:55.967212 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" event={"ID":"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0","Type":"ContainerStarted","Data":"d5b7e977af758a09273c1b94acdce28e1989f8a16d3dffbdadb53159af870a8f"} Oct 31 01:01:56.978058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:56.978011 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" event={"ID":"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0","Type":"ContainerStarted","Data":"fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108"} Oct 31 01:01:56.978847 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:56.978160 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:01:57.092929 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:01:57.092872 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" podStartSLOduration=2.136029081 podStartE2EDuration="3.092858811s" totalImagesPullingTime="956.82973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:57.091487743 +0000 UTC m=+2009.293738175" watchObservedRunningTime="2026-10-31 01:01:57.092858811 +0000 UTC m=+2009.295109244" Oct 31 01:02:02.993334 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:02.993301 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:02:07.018883 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:02:07.018843 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:02:28.283158 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:28.283118 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv"] Oct 31 01:02:28.283687 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:28.283373 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" containerID="cri-o://2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39" gracePeriod=30 Oct 31 01:02:29.042620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:29.042587 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:02:29.042743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:29.042643 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:02:29.897892 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:29.897851 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb"] Oct 31 01:02:29.899136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:29.899097 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" podUID="b6b7291a-105a-4cee-9d55-a5ae80e1e571" containerName="payload-processing" containerID="cri-o://91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899" gracePeriod=30 Oct 31 01:02:30.023139 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.023107 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb"] Oct 31 01:02:30.028216 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.028156 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" containerID="cri-o://9d1adc88be2ee495ca6a4abef1666fbd258c4298daec03b55a218ab0c3829b1f" gracePeriod=30 Oct 31 01:02:30.116593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.116559 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b"] Oct 31 01:02:30.116841 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.116811 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" containerID="cri-o://807a0c7bd0facd8384cf69283db924ad460390ac6a0364d2d25cbdc7863f63c1" gracePeriod=30 Oct 31 01:02:30.228497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.228466 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 01:02:30.253559 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.253522 5890 generic.go:356] "Generic (PLEG): container finished" podID="b6b7291a-105a-4cee-9d55-a5ae80e1e571" containerID="91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899" exitCode=0 Oct 31 01:02:30.253722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.253568 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" event={"ID":"b6b7291a-105a-4cee-9d55-a5ae80e1e571","Type":"ContainerDied","Data":"91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899"} Oct 31 01:02:30.253722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.253580 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" Oct 31 01:02:30.253722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.253621 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb" event={"ID":"b6b7291a-105a-4cee-9d55-a5ae80e1e571","Type":"ContainerDied","Data":"fd567c3ffc3294e22661175be79b9f6f0d11f9e39d782a1e45734f68bc8155b2"} Oct 31 01:02:30.253722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.253653 5890 scope.go:122] "RemoveContainer" containerID="91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899" Oct 31 01:02:30.264644 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.264570 5890 scope.go:122] "RemoveContainer" containerID="91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899" Oct 31 01:02:30.264896 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:02:30.264873 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899\": container with ID starting with 91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899 not found: ID does not exist" containerID="91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899" Oct 31 01:02:30.264977 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.264906 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899"} err="failed to get container status \"91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899\": rpc error: code = NotFound desc = could not find container \"91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899\": container with ID starting with 91bdf50e656731833e3f73017a5960a76f9dea9c3302ef6f022d46f9de9d4899 not found: ID does not exist" Oct 31 01:02:30.346287 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.346251 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b7291a-105a-4cee-9d55-a5ae80e1e571-kube-api-access-fkrl7\" (UniqueName: \"kubernetes.io/projected/b6b7291a-105a-4cee-9d55-a5ae80e1e571-kube-api-access-fkrl7\") pod \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " Oct 31 01:02:30.346489 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.346348 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b7291a-105a-4cee-9d55-a5ae80e1e571-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b7291a-105a-4cee-9d55-a5ae80e1e571-plugins-config-volume\") pod \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\" (UID: \"b6b7291a-105a-4cee-9d55-a5ae80e1e571\") " Oct 31 01:02:30.346760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.346733 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b7291a-105a-4cee-9d55-a5ae80e1e571-plugins-config-volume" pod "b6b7291a-105a-4cee-9d55-a5ae80e1e571" (UID: "b6b7291a-105a-4cee-9d55-a5ae80e1e571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:30.347117 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.347095 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b7291a-105a-4cee-9d55-a5ae80e1e571-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:30.348451 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.348430 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b7291a-105a-4cee-9d55-a5ae80e1e571-kube-api-access-fkrl7" pod "b6b7291a-105a-4cee-9d55-a5ae80e1e571" (UID: "b6b7291a-105a-4cee-9d55-a5ae80e1e571"). InnerVolumeSpecName "kube-api-access-fkrl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:30.455858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.451588 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkrl7\" (UniqueName: \"kubernetes.io/projected/b6b7291a-105a-4cee-9d55-a5ae80e1e571-kube-api-access-fkrl7\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:30.674195 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.674159 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb"] Oct 31 01:02:30.679861 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:30.679834 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ac952a-5b9c6cc7cb-2jbmb"] Oct 31 01:02:31.933440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:31.933407 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b7291a-105a-4cee-9d55-a5ae80e1e571" path="/var/lib/kubelet/pods/b6b7291a-105a-4cee-9d55-a5ae80e1e571/volumes" Oct 31 01:02:32.632270 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:32.632231 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-z2grb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:02:32.632444 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:32.632302 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:02:33.572848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.572818 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 01:02:33.707124 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707091 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-envoy\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707159 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80500604-236e-4c6c-8ffc-51add6824497-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80500604-236e-4c6c-8ffc-51add6824497-istiod-ca-cert\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707197 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-socket\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707225 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80500604-236e-4c6c-8ffc-51add6824497-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80500604-236e-4c6c-8ffc-51add6824497-istio-podinfo\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707294 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-data\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707331 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-certs\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707385 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-istio-token\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-istio-token\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707498 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-credential-socket\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707517 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-socket" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:33.707622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707565 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-kube-api-access-6lpn5\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-kube-api-access-6lpn5\") pod \"80500604-236e-4c6c-8ffc-51add6824497\" (UID: \"80500604-236e-4c6c-8ffc-51add6824497\") " Oct 31 01:02:33.707622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707566 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80500604-236e-4c6c-8ffc-51add6824497-istiod-ca-cert" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:33.707937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707703 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-data" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:33.707937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707717 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-credential-socket" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:33.708046 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.707952 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-certs" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:33.708210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.708181 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.708282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.708211 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80500604-236e-4c6c-8ffc-51add6824497-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.708282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.708226 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.708282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.708241 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.708282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.708254 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.709636 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.709612 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-istio-token" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:33.709825 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.709808 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-kube-api-access-6lpn5" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "kube-api-access-6lpn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:33.709923 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.709909 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-envoy" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:33.709961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.709934 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80500604-236e-4c6c-8ffc-51add6824497-istio-podinfo" pod "80500604-236e-4c6c-8ffc-51add6824497" (UID: "80500604-236e-4c6c-8ffc-51add6824497"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:33.809562 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.809478 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpn5\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-kube-api-access-6lpn5\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.809562 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.809508 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80500604-236e-4c6c-8ffc-51add6824497-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.809562 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.809519 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80500604-236e-4c6c-8ffc-51add6824497-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:33.809562 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:33.809527 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80500604-236e-4c6c-8ffc-51add6824497-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:34.287506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.287464 5890 generic.go:356] "Generic (PLEG): container finished" podID="80500604-236e-4c6c-8ffc-51add6824497" containerID="2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39" exitCode=0 Oct 31 01:02:34.287823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.287514 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" Oct 31 01:02:34.287823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.287515 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" event={"ID":"80500604-236e-4c6c-8ffc-51add6824497","Type":"ContainerDied","Data":"2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39"} Oct 31 01:02:34.287823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.287591 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv" event={"ID":"80500604-236e-4c6c-8ffc-51add6824497","Type":"ContainerDied","Data":"4a42493b47c5099a99f40a0c0c2e71a219b2e638ae36411954326a202634c5ac"} Oct 31 01:02:34.287823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.287623 5890 scope.go:122] "RemoveContainer" containerID="2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39" Oct 31 01:02:34.298207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.298188 5890 scope.go:122] "RemoveContainer" containerID="2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39" Oct 31 01:02:34.298479 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:02:34.298460 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39\": container with ID starting with 2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39 not found: ID does not exist" containerID="2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39" Oct 31 01:02:34.298553 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.298490 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39"} err="failed to get container status \"2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39\": rpc error: code = NotFound desc = could not find container \"2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39\": container with ID starting with 2354d6c80407cdc21b0f57ef975c01e06ac012521f5ece2d12902cccbee0ab39 not found: ID does not exist" Oct 31 01:02:34.496939 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.496904 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv"] Oct 31 01:02:34.545584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:34.545502 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-4f8rv"] Oct 31 01:02:35.306570 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.305910 5890 generic.go:356] "Generic (PLEG): container finished" podID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerID="807a0c7bd0facd8384cf69283db924ad460390ac6a0364d2d25cbdc7863f63c1" exitCode=0 Oct 31 01:02:35.306570 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.305969 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" event={"ID":"a1da0533-8d2e-4d41-aa09-10c4f66e84a4","Type":"ContainerDied","Data":"807a0c7bd0facd8384cf69283db924ad460390ac6a0364d2d25cbdc7863f63c1"} Oct 31 01:02:35.307842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.307815 5890 generic.go:356] "Generic (PLEG): container finished" podID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerID="9d1adc88be2ee495ca6a4abef1666fbd258c4298daec03b55a218ab0c3829b1f" exitCode=0 Oct 31 01:02:35.307965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.307882 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" event={"ID":"fdddbf4a-a26a-4d40-93d6-accc27dc93eb","Type":"ContainerDied","Data":"9d1adc88be2ee495ca6a4abef1666fbd258c4298daec03b55a218ab0c3829b1f"} Oct 31 01:02:35.494135 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.494109 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 01:02:35.524528 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.524500 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 01:02:35.531819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.531795 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-envoy\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.531946 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.531840 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-socket\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.531946 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.531872 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-credential-socket\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.531946 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.531908 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-credential-socket\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.531946 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.531929 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-token\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-token\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.532165 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.531957 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-socket\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.532165 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.531984 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istiod-ca-cert\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.532165 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532007 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-data\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.532165 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532058 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-kube-api-access-ptpmv\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-kube-api-access-ptpmv\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.532165 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532110 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istiod-ca-cert\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.532165 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532148 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-socket" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532169 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-credential-socket" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532177 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-credential-socket" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532183 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-podinfo\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532259 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-token\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-token\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532276 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-socket" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532300 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-envoy\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532338 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-certs\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532380 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-data" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.532477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532451 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-data\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.533004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532492 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-kube-api-access-r7rtk\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-kube-api-access-r7rtk\") pod \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\" (UID: \"a1da0533-8d2e-4d41-aa09-10c4f66e84a4\") " Oct 31 01:02:35.533004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532528 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istiod-ca-cert" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:35.533004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532560 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-podinfo\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.533004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532572 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istiod-ca-cert" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:35.533004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532594 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-certs\") pod \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\" (UID: \"fdddbf4a-a26a-4d40-93d6-accc27dc93eb\") " Oct 31 01:02:35.533004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532775 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-certs" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.533004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.532978 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-certs" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533160 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-data" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533264 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533283 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533297 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533312 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533326 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533340 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533354 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533368 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533837 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533381 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.533837 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.533393 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.534555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.534528 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-envoy" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.535009 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.534978 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-token" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:35.535148 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.535101 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-podinfo" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:35.535211 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.535152 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-kube-api-access-ptpmv" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "kube-api-access-ptpmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:35.535423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.535404 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-podinfo" pod "fdddbf4a-a26a-4d40-93d6-accc27dc93eb" (UID: "fdddbf4a-a26a-4d40-93d6-accc27dc93eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:35.535516 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.535494 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-envoy" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.535957 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.535932 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-kube-api-access-r7rtk" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "kube-api-access-r7rtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:35.536458 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.536441 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-token" pod "a1da0533-8d2e-4d41-aa09-10c4f66e84a4" (UID: "a1da0533-8d2e-4d41-aa09-10c4f66e84a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:35.635004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.634925 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.635004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.634963 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.635004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.634979 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.635004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.634993 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptpmv\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-kube-api-access-ptpmv\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.635265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.635012 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.635265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.635028 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdddbf4a-a26a-4d40-93d6-accc27dc93eb-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.635265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.635041 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.635265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.635063 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rtk\" (UniqueName: \"kubernetes.io/projected/a1da0533-8d2e-4d41-aa09-10c4f66e84a4-kube-api-access-r7rtk\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.934533 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:35.934495 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80500604-236e-4c6c-8ffc-51add6824497" path="/var/lib/kubelet/pods/80500604-236e-4c6c-8ffc-51add6824497/volumes" Oct 31 01:02:36.316774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.316695 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" Oct 31 01:02:36.316774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.316708 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb" event={"ID":"fdddbf4a-a26a-4d40-93d6-accc27dc93eb","Type":"ContainerDied","Data":"3efa059e130bb28cd746528a921a5aa31afc4fe96964c5918fd2e5a51404fe93"} Oct 31 01:02:36.316774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.316773 5890 scope.go:122] "RemoveContainer" containerID="9d1adc88be2ee495ca6a4abef1666fbd258c4298daec03b55a218ab0c3829b1f" Oct 31 01:02:36.318581 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.318544 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" event={"ID":"a1da0533-8d2e-4d41-aa09-10c4f66e84a4","Type":"ContainerDied","Data":"697e88e609d38d3c45251fe866f75777e45953e4e5d759b839a7a0ed90b4b2f0"} Oct 31 01:02:36.318718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.318550 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b" Oct 31 01:02:36.330151 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.330126 5890 scope.go:122] "RemoveContainer" containerID="807a0c7bd0facd8384cf69283db924ad460390ac6a0364d2d25cbdc7863f63c1" Oct 31 01:02:36.526025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.525992 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b"] Oct 31 01:02:36.573951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.573857 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-ckw6b"] Oct 31 01:02:36.636068 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.636031 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb"] Oct 31 01:02:36.639257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:36.639220 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-z2grb"] Oct 31 01:02:37.020368 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:02:37.020329 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:02:37.934058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:37.934013 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" path="/var/lib/kubelet/pods/a1da0533-8d2e-4d41-aa09-10c4f66e84a4/volumes" Oct 31 01:02:37.934641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:37.934621 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" path="/var/lib/kubelet/pods/fdddbf4a-a26a-4d40-93d6-accc27dc93eb/volumes" Oct 31 01:02:43.638527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.638487 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw"] Oct 31 01:02:43.639133 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639116 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" Oct 31 01:02:43.639181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639133 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" Oct 31 01:02:43.639181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639162 5890 cpu_manager.go:414] "containerMap: removing container" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" Oct 31 01:02:43.639181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639178 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" Oct 31 01:02:43.639283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639202 5890 cpu_manager.go:414] "containerMap: removing container" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" Oct 31 01:02:43.639283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639207 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" Oct 31 01:02:43.639283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639214 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b6b7291a-105a-4cee-9d55-a5ae80e1e571" containerName="payload-processing" Oct 31 01:02:43.639283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639219 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b7291a-105a-4cee-9d55-a5ae80e1e571" containerName="payload-processing" Oct 31 01:02:43.639401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639311 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdddbf4a-a26a-4d40-93d6-accc27dc93eb" containerName="istio-proxy" Oct 31 01:02:43.639401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639321 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b7291a-105a-4cee-9d55-a5ae80e1e571" containerName="payload-processing" Oct 31 01:02:43.639401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639329 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1da0533-8d2e-4d41-aa09-10c4f66e84a4" containerName="istio-proxy" Oct 31 01:02:43.639401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.639339 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="80500604-236e-4c6c-8ffc-51add6824497" containerName="istio-proxy" Oct 31 01:02:43.644254 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.644232 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.698540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.698509 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw"] Oct 31 01:02:43.742609 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.742578 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9e4801-serving-cert\"" Oct 31 01:02:43.748288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.748254 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02152a8e-0532-46e2-802e-6e711b205881-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.748440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.748392 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.748491 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.748454 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rzr\" (UniqueName: \"kubernetes.io/projected/02152a8e-0532-46e2-802e-6e711b205881-kube-api-access-n5rzr\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.748491 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.748485 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-metrics-certs\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.849701 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.849660 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.849916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.849720 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rzr\" (UniqueName: \"kubernetes.io/projected/02152a8e-0532-46e2-802e-6e711b205881-kube-api-access-n5rzr\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.849916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.849766 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-metrics-certs\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.849916 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:02:43.849823 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-serving-cert: secret "maas-api-e2e-praxis-b-9e4801-serving-cert" not found Oct 31 01:02:43.849916 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:02:43.849904 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls podName:02152a8e-0532-46e2-802e-6e711b205881 nodeName:}" failed. No retries permitted until 2026-10-31 01:02:44.349882372 +0000 UTC m=+2056.552132800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls") pod "maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" (UID: "02152a8e-0532-46e2-802e-6e711b205881") : secret "maas-api-e2e-praxis-b-9e4801-serving-cert" not found Oct 31 01:02:43.850160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.850005 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02152a8e-0532-46e2-802e-6e711b205881-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.850649 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.850625 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02152a8e-0532-46e2-802e-6e711b205881-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.852868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.852840 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-metrics-certs\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:43.912237 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:43.912164 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rzr\" (UniqueName: \"kubernetes.io/projected/02152a8e-0532-46e2-802e-6e711b205881-kube-api-access-n5rzr\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:44.357416 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:44.357379 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:44.360089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:44.360056 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:44.562250 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:44.562213 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:44.938726 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:44.938693 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw"] Oct 31 01:02:44.939532 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:02:44.939505 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02152a8e_0532_46e2_802e_6e711b205881.slice/crio-1fab49da4dca5d51391c327e45ead6c8420931126bff0bd59e76a40eefe909ea WatchSource:0}: Error finding container 1fab49da4dca5d51391c327e45ead6c8420931126bff0bd59e76a40eefe909ea: Status 404 returned error can't find the container with id 1fab49da4dca5d51391c327e45ead6c8420931126bff0bd59e76a40eefe909ea Oct 31 01:02:44.941048 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:44.941027 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:02:44.995796 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:44.995756 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9"] Oct 31 01:02:44.995996 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:44.995973 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" containerID="cri-o://fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772" gracePeriod=30 Oct 31 01:02:45.395318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:45.395223 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" event={"ID":"02152a8e-0532-46e2-802e-6e711b205881","Type":"ContainerStarted","Data":"1fab49da4dca5d51391c327e45ead6c8420931126bff0bd59e76a40eefe909ea"} Oct 31 01:02:46.403758 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:46.403716 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" event={"ID":"02152a8e-0532-46e2-802e-6e711b205881","Type":"ContainerStarted","Data":"c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168"} Oct 31 01:02:46.404185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:46.403838 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:02:46.517403 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:46.517306 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" podStartSLOduration=2.2859114959999998 podStartE2EDuration="3.51729428s" totalImagesPullingTime="1.231382784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:46.516719419 +0000 UTC m=+2058.718969851" watchObservedRunningTime="2026-10-31 01:02:46.51729428 +0000 UTC m=+2058.719544713" Oct 31 01:02:50.281820 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.281778 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438476 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-socket\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438538 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-envoy\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438573 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-certs\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438647 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-data\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438687 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-token\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-token\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438721 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-podinfo\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438846 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istiod-ca-cert\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438918 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-kube-api-access-p9wwc\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-kube-api-access-p9wwc\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.438960 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-credential-socket\") pod \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\" (UID: \"f73c23d4-bcc2-4b35-9d1d-4868db6698f5\") " Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.439969 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-credential-socket" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.440498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.440216 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-socket" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.443588 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.443558 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-certs" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.443807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.443638 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-data" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.443933 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.443675 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istiod-ca-cert" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:50.449457 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.449371 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-kube-api-access-p9wwc" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "kube-api-access-p9wwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:50.449674 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.449643 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-envoy" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.449938 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.449910 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-token" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:50.450841 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.450758 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-podinfo" pod "f73c23d4-bcc2-4b35-9d1d-4868db6698f5" (UID: "f73c23d4-bcc2-4b35-9d1d-4868db6698f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:50.466819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.459402 5890 generic.go:356] "Generic (PLEG): container finished" podID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerID="fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772" exitCode=0 Oct 31 01:02:50.466819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.459532 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" Oct 31 01:02:50.466819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.459664 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" event={"ID":"f73c23d4-bcc2-4b35-9d1d-4868db6698f5","Type":"ContainerDied","Data":"fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772"} Oct 31 01:02:50.466819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.459748 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9" event={"ID":"f73c23d4-bcc2-4b35-9d1d-4868db6698f5","Type":"ContainerDied","Data":"94b831a8b6144d5c06e30418ebafcf339b6cddbbb15a1a943c2a8d1d1427458d"} Oct 31 01:02:50.466819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.459800 5890 scope.go:122] "RemoveContainer" containerID="fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772" Oct 31 01:02:50.490584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.490560 5890 scope.go:122] "RemoveContainer" containerID="fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772" Oct 31 01:02:50.491042 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:02:50.491013 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772\": container with ID starting with fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772 not found: ID does not exist" containerID="fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772" Oct 31 01:02:50.491143 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.491057 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772"} err="failed to get container status \"fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772\": rpc error: code = NotFound desc = could not find container \"fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772\": container with ID starting with fd8a70964e00e8e237d2fdeb9bfc59405346172ac1c3780b399231c80b4ec772 not found: ID does not exist" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.540929 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.540957 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.540968 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.540977 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9wwc\" (UniqueName: \"kubernetes.io/projected/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-kube-api-access-p9wwc\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.540987 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.540996 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.541006 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.541016 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.541374 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.541026 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73c23d4-bcc2-4b35-9d1d-4868db6698f5-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.619372 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.619337 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9"] Oct 31 01:02:50.625294 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:50.625262 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-p9kw9"] Oct 31 01:02:51.933962 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:51.933930 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" path="/var/lib/kubelet/pods/f73c23d4-bcc2-4b35-9d1d-4868db6698f5/volumes" Oct 31 01:02:52.415866 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:02:52.415759 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:03:07.019186 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:03:07.019149 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:03:14.658743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:14.658702 5890 generic.go:356] "Generic (PLEG): container finished" podID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerID="63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657" exitCode=0 Oct 31 01:03:14.659204 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:14.658772 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" event={"ID":"50379ab5-7dc7-4d22-9ea8-194e4be2aba5","Type":"ContainerDied","Data":"63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657"} Oct 31 01:03:14.659263 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:14.659255 5890 scope.go:122] "RemoveContainer" containerID="63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657" Oct 31 01:03:14.660870 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:14.660841 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:03:14.660965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:14.660910 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:03:14.660965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:14.660944 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:03:15.674621 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:15.674585 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" event={"ID":"50379ab5-7dc7-4d22-9ea8-194e4be2aba5","Type":"ContainerStarted","Data":"7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e"} Oct 31 01:03:15.784584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:15.784551 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:03:15.784749 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:15.784722 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 01:03:15.784829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:15.784770 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 01:03:16.785141 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:16.785106 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 01:03:16.785530 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:16.785175 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 01:03:17.784665 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:17.784629 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 01:03:17.784862 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:17.784690 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 01:03:18.784589 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:18.784550 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 01:03:18.785219 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:18.785166 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 01:03:19.784437 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:19.784399 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 01:03:19.784620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:19.784470 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 01:03:20.862029 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:20.861998 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:03:20.863112 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:20.863085 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:03:20.864068 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:20.864046 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:03:21.504872 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:21.504834 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm"] Oct 31 01:03:21.505129 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:21.505104 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" containerID="cri-o://e228b0e08592371c30cc27e328bccdd9ffbadd9744ed6badb9ecf66d26e4b171" gracePeriod=30 Oct 31 01:03:25.975943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:25.975903 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj"] Oct 31 01:03:26.759638 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.759609 5890 generic.go:356] "Generic (PLEG): container finished" podID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerID="e228b0e08592371c30cc27e328bccdd9ffbadd9744ed6badb9ecf66d26e4b171" exitCode=0 Oct 31 01:03:26.759776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.759682 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" event={"ID":"b64b373e-2051-4517-b1c0-a25c5f61d217","Type":"ContainerDied","Data":"e228b0e08592371c30cc27e328bccdd9ffbadd9744ed6badb9ecf66d26e4b171"} Oct 31 01:03:26.759776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.759756 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" event={"ID":"b64b373e-2051-4517-b1c0-a25c5f61d217","Type":"ContainerDied","Data":"423f1b23056767b30f6f5643cbd797a2c0b7639b00dd291f83039a11513fd833"} Oct 31 01:03:26.759903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.759818 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="423f1b23056767b30f6f5643cbd797a2c0b7639b00dd291f83039a11513fd833" Oct 31 01:03:26.759903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.759835 5890 scope.go:122] "RemoveContainer" containerID="52cc85aae700c11dc46b80e04f7ce29e6a96151b99525447bf4a1bcf655c59a7" Oct 31 01:03:26.783180 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.783156 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 01:03:26.878943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.878912 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-podinfo\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879127 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.878985 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-envoy\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879127 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879013 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-socket\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879246 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879129 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-credential-socket\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879246 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879166 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-certs\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879246 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879204 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-data\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879409 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879257 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-token\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-token\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879409 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879294 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-kube-api-access-fplf2\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-kube-api-access-fplf2\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879409 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879375 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64b373e-2051-4517-b1c0-a25c5f61d217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b373e-2051-4517-b1c0-a25c5f61d217-istiod-ca-cert\") pod \"b64b373e-2051-4517-b1c0-a25c5f61d217\" (UID: \"b64b373e-2051-4517-b1c0-a25c5f61d217\") " Oct 31 01:03:26.879576 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879432 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-credential-socket" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:26.879576 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879453 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-certs" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:26.879698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879646 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-data" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:26.879846 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.879753 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-socket" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:26.880088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.880030 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64b373e-2051-4517-b1c0-a25c5f61d217-istiod-ca-cert" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:26.880181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.880090 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.880181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.880112 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.880181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.880128 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.880181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.880142 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.881367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.881341 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-podinfo" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:03:26.881672 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.881648 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-kube-api-access-fplf2" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "kube-api-access-fplf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:26.881779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.881748 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-envoy" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:26.881779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.881745 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-token" pod "b64b373e-2051-4517-b1c0-a25c5f61d217" (UID: "b64b373e-2051-4517-b1c0-a25c5f61d217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:26.981736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.981690 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.981736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.981735 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.982196 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.981751 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.982196 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.981767 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fplf2\" (UniqueName: \"kubernetes.io/projected/b64b373e-2051-4517-b1c0-a25c5f61d217-kube-api-access-fplf2\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:26.982196 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:26.981807 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b373e-2051-4517-b1c0-a25c5f61d217-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:27.767799 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:27.767755 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm" Oct 31 01:03:28.050029 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:28.049944 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm"] Oct 31 01:03:28.066622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:28.066583 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ac952a-openshift-default-7c54dc7589-gpplm"] Oct 31 01:03:28.775837 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:28.775801 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:03:28.896223 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:28.896198 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:03:29.062927 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:29.062838 5890 scope.go:122] "RemoveContainer" containerID="e228b0e08592371c30cc27e328bccdd9ffbadd9744ed6badb9ecf66d26e4b171" Oct 31 01:03:29.933927 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:29.933893 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" path="/var/lib/kubelet/pods/b64b373e-2051-4517-b1c0-a25c5f61d217/volumes" Oct 31 01:03:30.976473 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:30.976395 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" podUID="25e37385-65a3-46b9-9389-b8525d69727a" containerName="payload-pre-processing" containerID="cri-o://8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8" gracePeriod=25 Oct 31 01:03:31.276113 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.276077 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 01:03:31.342025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.341994 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e37385-65a3-46b9-9389-b8525d69727a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e37385-65a3-46b9-9389-b8525d69727a-plugins-config-volume\") pod \"25e37385-65a3-46b9-9389-b8525d69727a\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " Oct 31 01:03:31.342196 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.342147 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e37385-65a3-46b9-9389-b8525d69727a-kube-api-access-lxdln\" (UniqueName: \"kubernetes.io/projected/25e37385-65a3-46b9-9389-b8525d69727a-kube-api-access-lxdln\") pod \"25e37385-65a3-46b9-9389-b8525d69727a\" (UID: \"25e37385-65a3-46b9-9389-b8525d69727a\") " Oct 31 01:03:31.342451 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.342415 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e37385-65a3-46b9-9389-b8525d69727a-plugins-config-volume" pod "25e37385-65a3-46b9-9389-b8525d69727a" (UID: "25e37385-65a3-46b9-9389-b8525d69727a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:31.344283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.344261 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e37385-65a3-46b9-9389-b8525d69727a-kube-api-access-lxdln" pod "25e37385-65a3-46b9-9389-b8525d69727a" (UID: "25e37385-65a3-46b9-9389-b8525d69727a"). InnerVolumeSpecName "kube-api-access-lxdln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:31.443729 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.443691 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e37385-65a3-46b9-9389-b8525d69727a-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:31.443729 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.443724 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxdln\" (UniqueName: \"kubernetes.io/projected/25e37385-65a3-46b9-9389-b8525d69727a-kube-api-access-lxdln\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:31.803168 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.803129 5890 generic.go:356] "Generic (PLEG): container finished" podID="25e37385-65a3-46b9-9389-b8525d69727a" containerID="8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8" exitCode=0 Oct 31 01:03:31.803355 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.803176 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" Oct 31 01:03:31.803355 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.803208 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" event={"ID":"25e37385-65a3-46b9-9389-b8525d69727a","Type":"ContainerDied","Data":"8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8"} Oct 31 01:03:31.803355 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.803294 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj" event={"ID":"25e37385-65a3-46b9-9389-b8525d69727a","Type":"ContainerDied","Data":"f1681392701325232d306482053b71c0b49e6effb2cc7dbec6e35c2ffee12088"} Oct 31 01:03:31.803355 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.803322 5890 scope.go:122] "RemoveContainer" containerID="8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8" Oct 31 01:03:31.814037 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.814013 5890 scope.go:122] "RemoveContainer" containerID="8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8" Oct 31 01:03:31.814330 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:03:31.814308 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8\": container with ID starting with 8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8 not found: ID does not exist" containerID="8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8" Oct 31 01:03:31.814401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.814343 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8"} err="failed to get container status \"8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8\": rpc error: code = NotFound desc = could not find container \"8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8\": container with ID starting with 8c5740dfcd99354ca899d9f425bad405d476e0c91b01341edb53832e4e14dab8 not found: ID does not exist" Oct 31 01:03:31.963199 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.963163 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj"] Oct 31 01:03:31.973668 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:31.973636 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eee7f1-b9f89c67f-kx6wj"] Oct 31 01:03:33.933035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:33.933003 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e37385-65a3-46b9-9389-b8525d69727a" path="/var/lib/kubelet/pods/25e37385-65a3-46b9-9389-b8525d69727a/volumes" Oct 31 01:03:37.020530 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:03:37.020488 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:03:39.945916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:39.945875 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v"] Oct 31 01:03:39.947730 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:39.947670 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" podUID="6285afa6-04d6-456a-a7f8-9fe220d34e5b" containerName="maas-api" containerID="cri-o://4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9" gracePeriod=30 Oct 31 01:03:40.237965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.237935 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:03:40.363950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.352465 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-maas-api-tls\") pod \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " Oct 31 01:03:40.363950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.352535 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-metrics-certs\") pod \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " Oct 31 01:03:40.363950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.352634 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6285afa6-04d6-456a-a7f8-9fe220d34e5b-kube-api-access-wqbl6\" (UniqueName: \"kubernetes.io/projected/6285afa6-04d6-456a-a7f8-9fe220d34e5b-kube-api-access-wqbl6\") pod \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " Oct 31 01:03:40.363950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.352809 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6285afa6-04d6-456a-a7f8-9fe220d34e5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6285afa6-04d6-456a-a7f8-9fe220d34e5b-openshift-service-ca\") pod \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\" (UID: \"6285afa6-04d6-456a-a7f8-9fe220d34e5b\") " Oct 31 01:03:40.363950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.353758 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6285afa6-04d6-456a-a7f8-9fe220d34e5b-openshift-service-ca" pod "6285afa6-04d6-456a-a7f8-9fe220d34e5b" (UID: "6285afa6-04d6-456a-a7f8-9fe220d34e5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:40.369739 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.367481 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6285afa6-04d6-456a-a7f8-9fe220d34e5b-kube-api-access-wqbl6" pod "6285afa6-04d6-456a-a7f8-9fe220d34e5b" (UID: "6285afa6-04d6-456a-a7f8-9fe220d34e5b"). InnerVolumeSpecName "kube-api-access-wqbl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:40.389965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.389923 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-metrics-certs" pod "6285afa6-04d6-456a-a7f8-9fe220d34e5b" (UID: "6285afa6-04d6-456a-a7f8-9fe220d34e5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:40.391740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.390943 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-maas-api-tls" pod "6285afa6-04d6-456a-a7f8-9fe220d34e5b" (UID: "6285afa6-04d6-456a-a7f8-9fe220d34e5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:40.454421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.454369 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.454421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.454414 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6285afa6-04d6-456a-a7f8-9fe220d34e5b-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.454421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.454428 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqbl6\" (UniqueName: \"kubernetes.io/projected/6285afa6-04d6-456a-a7f8-9fe220d34e5b-kube-api-access-wqbl6\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.454661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.454443 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6285afa6-04d6-456a-a7f8-9fe220d34e5b-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.869390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.869346 5890 generic.go:356] "Generic (PLEG): container finished" podID="6285afa6-04d6-456a-a7f8-9fe220d34e5b" containerID="4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9" exitCode=0 Oct 31 01:03:40.870103 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.869602 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" Oct 31 01:03:40.870251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.869622 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" event={"ID":"6285afa6-04d6-456a-a7f8-9fe220d34e5b","Type":"ContainerDied","Data":"4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9"} Oct 31 01:03:40.870251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.870218 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v" event={"ID":"6285afa6-04d6-456a-a7f8-9fe220d34e5b","Type":"ContainerDied","Data":"12efddc342415bacf1a621eb6c86b8097d57fca58f7d360f0b5c1abe479072d1"} Oct 31 01:03:40.870364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.870252 5890 scope.go:122] "RemoveContainer" containerID="4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9" Oct 31 01:03:40.885646 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.885625 5890 scope.go:122] "RemoveContainer" containerID="4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9" Oct 31 01:03:40.886013 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:03:40.885989 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9\": container with ID starting with 4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9 not found: ID does not exist" containerID="4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9" Oct 31 01:03:40.886115 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:40.886028 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9"} err="failed to get container status \"4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9\": rpc error: code = NotFound desc = could not find container \"4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9\": container with ID starting with 4e212b0d906e6c4bf470b9ff2e172f3983636884c6eadb9ca072a07c947e9cc9 not found: ID does not exist" Oct 31 01:03:41.031374 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:41.031338 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v"] Oct 31 01:03:41.058027 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:41.057987 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb41d3-5996f6c4d6-9288v"] Oct 31 01:03:41.933491 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:41.933454 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6285afa6-04d6-456a-a7f8-9fe220d34e5b" path="/var/lib/kubelet/pods/6285afa6-04d6-456a-a7f8-9fe220d34e5b/volumes" Oct 31 01:03:52.556269 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556230 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph"] Oct 31 01:03:52.556850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556832 5890 cpu_manager.go:414] "containerMap: removing container" podUID="25e37385-65a3-46b9-9389-b8525d69727a" containerName="payload-pre-processing" Oct 31 01:03:52.556903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556850 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e37385-65a3-46b9-9389-b8525d69727a" containerName="payload-pre-processing" Oct 31 01:03:52.556903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556870 5890 cpu_manager.go:414] "containerMap: removing container" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" Oct 31 01:03:52.556903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556875 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" Oct 31 01:03:52.556903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556885 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" Oct 31 01:03:52.556903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556890 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" Oct 31 01:03:52.556903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556901 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6285afa6-04d6-456a-a7f8-9fe220d34e5b" containerName="maas-api" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556905 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6285afa6-04d6-456a-a7f8-9fe220d34e5b" containerName="maas-api" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556921 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.556926 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.557022 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73c23d4-bcc2-4b35-9d1d-4868db6698f5" containerName="istio-proxy" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.557031 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.557036 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6285afa6-04d6-456a-a7f8-9fe220d34e5b" containerName="maas-api" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.557046 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64b373e-2051-4517-b1c0-a25c5f61d217" containerName="istio-proxy" Oct 31 01:03:52.557085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.557057 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e37385-65a3-46b9-9389-b8525d69727a" containerName="payload-pre-processing" Oct 31 01:03:52.559361 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.559339 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.619629 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.619595 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dph5w\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-kube-api-access-dph5w\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.619848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.619633 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-socket\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.619848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.619681 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-envoy\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.619982 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.619872 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-data\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.619982 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.619931 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-certs\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.620088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.619981 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-istio-token\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.620149 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.620123 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-credential-socket\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.620213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.620157 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbca718-dec3-44fc-a165-6c372866230b-istio-podinfo\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.620213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.620180 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbca718-dec3-44fc-a165-6c372866230b-istiod-ca-cert\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.630343 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.630311 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph"] Oct 31 01:03:52.688213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.688181 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-add5bf-openshift-default-dockercfg-b2h4p\"" Oct 31 01:03:52.693038 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.693000 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf"] Oct 31 01:03:52.693255 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.693233 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" podUID="74216060-038e-4445-b056-91e9283a8af8" containerName="payload-processing" containerID="cri-o://23cf2129db7900c6142c7a1bb72329444acee863b821f2260ff3eb269d28a58c" gracePeriod=30 Oct 31 01:03:52.721268 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721241 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-data\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721396 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721283 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-certs\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721396 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721303 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-istio-token\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721396 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721382 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-credential-socket\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721635 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721423 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbca718-dec3-44fc-a165-6c372866230b-istio-podinfo\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721635 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721446 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbca718-dec3-44fc-a165-6c372866230b-istiod-ca-cert\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721635 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721522 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dph5w\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-kube-api-access-dph5w\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721635 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721549 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-socket\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721635 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721608 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-envoy\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721720 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-data\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.721883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721870 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-credential-socket\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.722007 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721988 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-certs\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.722076 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.721996 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-socket\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.722352 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.722331 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbca718-dec3-44fc-a165-6c372866230b-istiod-ca-cert\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.723914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.723890 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-envoy\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.724166 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.724148 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbca718-dec3-44fc-a165-6c372866230b-istio-podinfo\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.747527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.747500 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg"] Oct 31 01:03:52.747715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.747694 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" podUID="2d3235a9-ace9-49c7-ae0a-5567ccacc120" containerName="payload-pre-processing" containerID="cri-o://6f13639916a5e1b3891548cc4f99291d315dcc5281496d325b49dbcab96bfc71" gracePeriod=30 Oct 31 01:03:52.783983 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.783952 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-istio-token\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.784941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.784915 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dph5w\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-kube-api-access-dph5w\") pod \"e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.871948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.871917 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:52.965491 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.964345 5890 generic.go:356] "Generic (PLEG): container finished" podID="74216060-038e-4445-b056-91e9283a8af8" containerID="23cf2129db7900c6142c7a1bb72329444acee863b821f2260ff3eb269d28a58c" exitCode=0 Oct 31 01:03:52.965491 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.964467 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" event={"ID":"74216060-038e-4445-b056-91e9283a8af8","Type":"ContainerDied","Data":"23cf2129db7900c6142c7a1bb72329444acee863b821f2260ff3eb269d28a58c"} Oct 31 01:03:52.967249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.966889 5890 generic.go:356] "Generic (PLEG): container finished" podID="2d3235a9-ace9-49c7-ae0a-5567ccacc120" containerID="6f13639916a5e1b3891548cc4f99291d315dcc5281496d325b49dbcab96bfc71" exitCode=0 Oct 31 01:03:52.967249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:52.967096 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" event={"ID":"2d3235a9-ace9-49c7-ae0a-5567ccacc120","Type":"ContainerDied","Data":"6f13639916a5e1b3891548cc4f99291d315dcc5281496d325b49dbcab96bfc71"} Oct 31 01:03:53.027823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.027141 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 01:03:53.078972 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.078939 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 01:03:53.127853 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.127756 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3235a9-ace9-49c7-ae0a-5567ccacc120-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3235a9-ace9-49c7-ae0a-5567ccacc120-plugins-config-volume\") pod \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " Oct 31 01:03:53.127853 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.127833 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74216060-038e-4445-b056-91e9283a8af8-kube-api-access-d2wdq\" (UniqueName: \"kubernetes.io/projected/74216060-038e-4445-b056-91e9283a8af8-kube-api-access-d2wdq\") pod \"74216060-038e-4445-b056-91e9283a8af8\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " Oct 31 01:03:53.128053 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.127997 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3235a9-ace9-49c7-ae0a-5567ccacc120-kube-api-access-gdj54\" (UniqueName: \"kubernetes.io/projected/2d3235a9-ace9-49c7-ae0a-5567ccacc120-kube-api-access-gdj54\") pod \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\" (UID: \"2d3235a9-ace9-49c7-ae0a-5567ccacc120\") " Oct 31 01:03:53.128053 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.128028 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74216060-038e-4445-b056-91e9283a8af8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74216060-038e-4445-b056-91e9283a8af8-plugins-config-volume\") pod \"74216060-038e-4445-b056-91e9283a8af8\" (UID: \"74216060-038e-4445-b056-91e9283a8af8\") " Oct 31 01:03:53.128229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.128203 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3235a9-ace9-49c7-ae0a-5567ccacc120-plugins-config-volume" pod "2d3235a9-ace9-49c7-ae0a-5567ccacc120" (UID: "2d3235a9-ace9-49c7-ae0a-5567ccacc120"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:53.128458 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.128434 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74216060-038e-4445-b056-91e9283a8af8-plugins-config-volume" pod "74216060-038e-4445-b056-91e9283a8af8" (UID: "74216060-038e-4445-b056-91e9283a8af8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:53.128597 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.128580 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3235a9-ace9-49c7-ae0a-5567ccacc120-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:53.128651 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.128603 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74216060-038e-4445-b056-91e9283a8af8-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:53.129957 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.129937 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74216060-038e-4445-b056-91e9283a8af8-kube-api-access-d2wdq" pod "74216060-038e-4445-b056-91e9283a8af8" (UID: "74216060-038e-4445-b056-91e9283a8af8"). InnerVolumeSpecName "kube-api-access-d2wdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:53.130015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.130000 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3235a9-ace9-49c7-ae0a-5567ccacc120-kube-api-access-gdj54" pod "2d3235a9-ace9-49c7-ae0a-5567ccacc120" (UID: "2d3235a9-ace9-49c7-ae0a-5567ccacc120"). InnerVolumeSpecName "kube-api-access-gdj54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:53.233800 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.233710 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdj54\" (UniqueName: \"kubernetes.io/projected/2d3235a9-ace9-49c7-ae0a-5567ccacc120-kube-api-access-gdj54\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:53.233800 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.233748 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wdq\" (UniqueName: \"kubernetes.io/projected/74216060-038e-4445-b056-91e9283a8af8-kube-api-access-d2wdq\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:53.363992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.363828 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph"] Oct 31 01:03:53.371877 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:03:53.371826 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbca718_dec3_44fc_a165_6c372866230b.slice/crio-03b6e8dcf7fa96e50a1f301427504575a170f686f8751a44675bddcd95b95625 WatchSource:0}: Error finding container 03b6e8dcf7fa96e50a1f301427504575a170f686f8751a44675bddcd95b95625: Status 404 returned error can't find the container with id 03b6e8dcf7fa96e50a1f301427504575a170f686f8751a44675bddcd95b95625 Oct 31 01:03:53.377262 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.377217 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:03:53.377381 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.377311 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:03:53.377381 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.377352 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:03:53.979798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.977259 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" Oct 31 01:03:53.979798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.977291 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf" event={"ID":"74216060-038e-4445-b056-91e9283a8af8","Type":"ContainerDied","Data":"d7a0553b6095a08acf0ceb163aa625e9ffb782b13b30aeebc9fd478aa2a11694"} Oct 31 01:03:53.979798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.977353 5890 scope.go:122] "RemoveContainer" containerID="23cf2129db7900c6142c7a1bb72329444acee863b821f2260ff3eb269d28a58c" Oct 31 01:03:53.979798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.979415 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" Oct 31 01:03:53.979798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.979467 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg" event={"ID":"2d3235a9-ace9-49c7-ae0a-5567ccacc120","Type":"ContainerDied","Data":"5d6af3458db9a9e55f7d604310d5319e574b7d0e37c6f186aada66b686577c33"} Oct 31 01:03:53.983801 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.981683 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" event={"ID":"ecbca718-dec3-44fc-a165-6c372866230b","Type":"ContainerStarted","Data":"7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a"} Oct 31 01:03:53.983801 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.981731 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" event={"ID":"ecbca718-dec3-44fc-a165-6c372866230b","Type":"ContainerStarted","Data":"03b6e8dcf7fa96e50a1f301427504575a170f686f8751a44675bddcd95b95625"} Oct 31 01:03:53.990928 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:53.990903 5890 scope.go:122] "RemoveContainer" containerID="6f13639916a5e1b3891548cc4f99291d315dcc5281496d325b49dbcab96bfc71" Oct 31 01:03:54.147206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.147173 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg"] Oct 31 01:03:54.157571 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.157545 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb2b8a-bb5bcdff5-d52gg"] Oct 31 01:03:54.320348 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.320245 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podStartSLOduration=2.320231884 podStartE2EDuration="2.320231884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:54.318616078 +0000 UTC m=+2126.520866514" watchObservedRunningTime="2026-10-31 01:03:54.320231884 +0000 UTC m=+2126.522482317" Oct 31 01:03:54.438305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.438272 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf"] Oct 31 01:03:54.458252 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.456960 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb2b8a-569f8b994d-4p5pf"] Oct 31 01:03:54.872916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.872880 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:54.878565 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.878536 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:54.991927 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.991897 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:54.992951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:54.992928 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:03:55.933701 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:55.933666 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3235a9-ace9-49c7-ae0a-5567ccacc120" path="/var/lib/kubelet/pods/2d3235a9-ace9-49c7-ae0a-5567ccacc120/volumes" Oct 31 01:03:55.934124 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:55.934110 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74216060-038e-4445-b056-91e9283a8af8" path="/var/lib/kubelet/pods/74216060-038e-4445-b056-91e9283a8af8/volumes" Oct 31 01:03:57.822407 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.822372 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l"] Oct 31 01:03:57.823021 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.823005 5890 cpu_manager.go:414] "containerMap: removing container" podUID="2d3235a9-ace9-49c7-ae0a-5567ccacc120" containerName="payload-pre-processing" Oct 31 01:03:57.823093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.823023 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3235a9-ace9-49c7-ae0a-5567ccacc120" containerName="payload-pre-processing" Oct 31 01:03:57.823093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.823038 5890 cpu_manager.go:414] "containerMap: removing container" podUID="74216060-038e-4445-b056-91e9283a8af8" containerName="payload-processing" Oct 31 01:03:57.823093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.823044 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74216060-038e-4445-b056-91e9283a8af8" containerName="payload-processing" Oct 31 01:03:57.823210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.823148 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3235a9-ace9-49c7-ae0a-5567ccacc120" containerName="payload-pre-processing" Oct 31 01:03:57.823210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.823159 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="74216060-038e-4445-b056-91e9283a8af8" containerName="payload-processing" Oct 31 01:03:57.827876 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.827857 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:57.881663 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.881631 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l"] Oct 31 01:03:57.902034 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.901995 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhngr\" (UniqueName: \"kubernetes.io/projected/3f75dd97-30d3-495c-ab40-6930d43163ff-kube-api-access-nhngr\") pod \"payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:57.902328 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.902300 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f75dd97-30d3-495c-ab40-6930d43163ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:57.925538 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.925511 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4ab4aa\"" Oct 31 01:03:57.925736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:57.925585 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4ab4aa-dockercfg-5xjw2\"" Oct 31 01:03:58.004152 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:58.004102 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f75dd97-30d3-495c-ab40-6930d43163ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:58.005130 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:58.005101 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhngr\" (UniqueName: \"kubernetes.io/projected/3f75dd97-30d3-495c-ab40-6930d43163ff-kube-api-access-nhngr\") pod \"payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:58.006027 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:58.005980 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f75dd97-30d3-495c-ab40-6930d43163ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:58.075890 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:58.075769 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhngr\" (UniqueName: \"kubernetes.io/projected/3f75dd97-30d3-495c-ab40-6930d43163ff-kube-api-access-nhngr\") pod \"payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:58.142106 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:58.142070 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:58.542231 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:58.542201 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l"] Oct 31 01:03:58.542844 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:03:58.542815 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f75dd97_30d3_495c_ab40_6930d43163ff.slice/crio-ebb79561b6283727a4ae213b692b976e9069da14ea35f3033227e8360026fa46 WatchSource:0}: Error finding container ebb79561b6283727a4ae213b692b976e9069da14ea35f3033227e8360026fa46: Status 404 returned error can't find the container with id ebb79561b6283727a4ae213b692b976e9069da14ea35f3033227e8360026fa46 Oct 31 01:03:59.030027 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:59.029988 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" event={"ID":"3f75dd97-30d3-495c-ab40-6930d43163ff","Type":"ContainerStarted","Data":"b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879"} Oct 31 01:03:59.030482 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:59.030068 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" event={"ID":"3f75dd97-30d3-495c-ab40-6930d43163ff","Type":"ContainerStarted","Data":"ebb79561b6283727a4ae213b692b976e9069da14ea35f3033227e8360026fa46"} Oct 31 01:03:59.030482 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:59.030118 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:03:59.156273 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:03:59.156212 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" podStartSLOduration=2.156194414 podStartE2EDuration="2.156194414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:59.150930316 +0000 UTC m=+2131.353180749" watchObservedRunningTime="2026-10-31 01:03:59.156194414 +0000 UTC m=+2131.358444856" Oct 31 01:04:02.676710 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.676674 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7"] Oct 31 01:04:02.681332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.681310 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:02.734735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.734701 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-add5bf-dockercfg-ffmd6\"" Oct 31 01:04:02.737446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.737415 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7"] Oct 31 01:04:02.781343 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.781311 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-add5bf\"" Oct 31 01:04:02.867045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.867010 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653bfba6-e4c6-43bb-995d-41c9be1c2033-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:02.867221 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.867125 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqns\" (UniqueName: \"kubernetes.io/projected/653bfba6-e4c6-43bb-995d-41c9be1c2033-kube-api-access-ckqns\") pod \"payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:02.969750 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.968127 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653bfba6-e4c6-43bb-995d-41c9be1c2033-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:02.969750 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.968933 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653bfba6-e4c6-43bb-995d-41c9be1c2033-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:02.969750 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:02.969082 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqns\" (UniqueName: \"kubernetes.io/projected/653bfba6-e4c6-43bb-995d-41c9be1c2033-kube-api-access-ckqns\") pod \"payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:03.029563 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:03.029526 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqns\" (UniqueName: \"kubernetes.io/projected/653bfba6-e4c6-43bb-995d-41c9be1c2033-kube-api-access-ckqns\") pod \"payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:03.297752 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:03.297664 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:03.685111 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:03.685084 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7"] Oct 31 01:04:03.685883 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:04:03.685852 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653bfba6_e4c6_43bb_995d_41c9be1c2033.slice/crio-c4a06c09fdaab5163f7a2eebe40ed4cbd4a14931b5d695b385fdb95f005e0c31 WatchSource:0}: Error finding container c4a06c09fdaab5163f7a2eebe40ed4cbd4a14931b5d695b385fdb95f005e0c31: Status 404 returned error can't find the container with id c4a06c09fdaab5163f7a2eebe40ed4cbd4a14931b5d695b385fdb95f005e0c31 Oct 31 01:04:04.064651 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:04.064568 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" event={"ID":"653bfba6-e4c6-43bb-995d-41c9be1c2033","Type":"ContainerStarted","Data":"54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7"} Oct 31 01:04:04.064651 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:04.064628 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" event={"ID":"653bfba6-e4c6-43bb-995d-41c9be1c2033","Type":"ContainerStarted","Data":"c4a06c09fdaab5163f7a2eebe40ed4cbd4a14931b5d695b385fdb95f005e0c31"} Oct 31 01:04:04.064989 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:04.064670 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:07.019428 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:04:07.019384 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:04:10.040666 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:10.040633 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:04:10.164438 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:10.164382 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" podStartSLOduration=8.164371139 podStartE2EDuration="8.164371139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:04.18488603 +0000 UTC m=+2136.387136464" watchObservedRunningTime="2026-10-31 01:04:10.164371139 +0000 UTC m=+2142.366621621" Oct 31 01:04:15.074502 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:15.074471 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:04:37.018057 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:04:37.018019 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:04:45.067527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.067487 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9"] Oct 31 01:04:45.074221 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.074193 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.137491 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.137309 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b63c24-openshift-default-dockercfg-58m59\"" Oct 31 01:04:45.192028 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.191993 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9"] Oct 31 01:04:45.214414 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.214368 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.214595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.214461 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.214595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.214521 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.214723 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.214651 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1e5e5e-249e-453b-92d1-cb12ab095404-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.214793 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.214758 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.214867 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.214847 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.215010 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.214987 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.215090 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.215029 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.215192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.215173 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2kq\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-kube-api-access-jz2kq\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316028 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2kq\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-kube-api-access-jz2kq\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316069 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316092 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316127 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316167 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1e5e5e-249e-453b-92d1-cb12ab095404-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316227 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316284 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316385 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316428 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316530 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316563 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316846 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316820 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.316987 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316814 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.317047 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.316979 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1e5e5e-249e-453b-92d1-cb12ab095404-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.318541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.318492 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.318834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.318815 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.391362 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.391317 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2kq\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-kube-api-access-jz2kq\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.392041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.392010 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:45.689827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:45.689771 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:46.136087 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:04:46.136048 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1e5e5e_249e_453b_92d1_cb12ab095404.slice/crio-2f8f0a4beea06f778f2b62c6981ce34b73751e57bdfd72f7170d067d23dc3b7f WatchSource:0}: Error finding container 2f8f0a4beea06f778f2b62c6981ce34b73751e57bdfd72f7170d067d23dc3b7f: Status 404 returned error can't find the container with id 2f8f0a4beea06f778f2b62c6981ce34b73751e57bdfd72f7170d067d23dc3b7f Oct 31 01:04:46.139058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.139015 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:04:46.139187 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.139111 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:04:46.139187 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.139148 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:04:46.139883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.139858 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9"] Oct 31 01:04:46.391224 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.391131 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" event={"ID":"2b1e5e5e-249e-453b-92d1-cb12ab095404","Type":"ContainerStarted","Data":"ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb"} Oct 31 01:04:46.391224 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.391205 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" event={"ID":"2b1e5e5e-249e-453b-92d1-cb12ab095404","Type":"ContainerStarted","Data":"2f8f0a4beea06f778f2b62c6981ce34b73751e57bdfd72f7170d067d23dc3b7f"} Oct 31 01:04:46.530347 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.530288 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podStartSLOduration=1.5302728 podStartE2EDuration="1.5302728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:46.52940615 +0000 UTC m=+2178.731656586" watchObservedRunningTime="2026-10-31 01:04:46.5302728 +0000 UTC m=+2178.732523233" Oct 31 01:04:46.690248 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:46.690208 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:47.692576 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:47.692536 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:04:47.693225 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:47.692617 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 01:04:48.691486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:48.691381 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:04:48.691827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:48.691772 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 01:04:49.690753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:49.690713 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:04:49.691197 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:49.690771 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 01:04:50.690908 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:50.690873 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:04:50.691314 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:50.690934 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 01:04:51.690661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:51.690628 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:04:51.690853 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:51.690684 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 01:04:52.691228 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:52.691185 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:04:52.691634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:52.691259 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 01:04:54.731943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:54.731906 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:04:54.732333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:54.731983 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:04:55.003619 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:55.003529 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:55.004029 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:55.004003 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:04:55.004858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:04:55.004837 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:05:07.028191 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:05:07.028141 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:05:35.764330 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:35.764294 5890 generic.go:356] "Generic (PLEG): container finished" podID="ecbca718-dec3-44fc-a165-6c372866230b" containerID="7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a" exitCode=0 Oct 31 01:05:35.764912 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:35.764372 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" event={"ID":"ecbca718-dec3-44fc-a165-6c372866230b","Type":"ContainerDied","Data":"7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a"} Oct 31 01:05:35.764912 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:35.764874 5890 scope.go:122] "RemoveContainer" containerID="7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a" Oct 31 01:05:35.766345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:35.766312 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:05:35.766454 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:35.766399 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:05:35.766519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:35.766449 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:05:36.775462 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:36.775421 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" event={"ID":"ecbca718-dec3-44fc-a165-6c372866230b","Type":"ContainerStarted","Data":"877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c"} Oct 31 01:05:36.872626 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:36.872583 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:05:36.874257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:36.874228 5890 patch_prober.go:29] interesting pod/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 01:05:36.874385 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:36.874278 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 01:05:37.018258 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:05:37.018216 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:05:37.873387 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:37.873349 5890 patch_prober.go:29] interesting pod/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 01:05:37.873848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:37.873412 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 01:05:38.873377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:38.873347 5890 patch_prober.go:29] interesting pod/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 01:05:38.873562 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:38.873397 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 01:05:39.875149 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:39.875104 5890 patch_prober.go:29] interesting pod/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 01:05:39.875555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:39.875168 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 01:05:39.992363 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:39.992327 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:05:40.872490 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:40.872453 5890 patch_prober.go:29] interesting pod/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 01:05:40.872676 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:40.872519 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 01:05:41.873766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:41.873725 5890 patch_prober.go:29] interesting pod/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 01:05:41.874205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:41.873811 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 01:05:42.877478 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:42.877448 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:05:42.878525 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:42.878503 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:05:44.999373 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:44.999339 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql"] Oct 31 01:05:45.002675 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.002656 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059140 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059190 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmt6z\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-kube-api-access-xmt6z\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059294 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059332 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a0f9e8-65df-441a-821e-952dca1a4641-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059479 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059359 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a0f9e8-65df-441a-821e-952dca1a4641-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059479 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059460 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059561 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059506 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059637 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059613 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.059735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.059719 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.105160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.105126 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql"] Oct 31 01:05:45.160555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160525 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.160751 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160594 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.160751 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160617 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmt6z\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-kube-api-access-xmt6z\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.160751 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160666 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.160751 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160720 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a0f9e8-65df-441a-821e-952dca1a4641-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160758 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a0f9e8-65df-441a-821e-952dca1a4641-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160810 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160862 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.160911 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161237 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.161010 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161365 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.161337 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161631 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.161612 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a0f9e8-65df-441a-821e-952dca1a4641-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161717 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.161627 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.161879 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.161859 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.163452 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.163426 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.163594 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.163570 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a0f9e8-65df-441a-821e-952dca1a4641-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.220088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.220060 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.220381 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.220361 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmt6z\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-kube-api-access-xmt6z\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.315994 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.315906 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:45.770125 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.770091 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql"] Oct 31 01:05:45.774463 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:05:45.774428 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a0f9e8_65df_441a_821e_952dca1a4641.slice/crio-dcb378dcb94b04ff66664ccc3b5c952ba8a48ae660979d057a9cd357fd02d1f5 WatchSource:0}: Error finding container dcb378dcb94b04ff66664ccc3b5c952ba8a48ae660979d057a9cd357fd02d1f5: Status 404 returned error can't find the container with id dcb378dcb94b04ff66664ccc3b5c952ba8a48ae660979d057a9cd357fd02d1f5 Oct 31 01:05:45.776989 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.776956 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:05:45.777057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.777029 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:05:45.777093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.777061 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:05:45.863525 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.863490 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" event={"ID":"08a0f9e8-65df-441a-821e-952dca1a4641","Type":"ContainerStarted","Data":"47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2"} Oct 31 01:05:45.863706 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.863571 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" event={"ID":"08a0f9e8-65df-441a-821e-952dca1a4641","Type":"ContainerStarted","Data":"dcb378dcb94b04ff66664ccc3b5c952ba8a48ae660979d057a9cd357fd02d1f5"} Oct 31 01:05:45.990040 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:45.989968 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podStartSLOduration=1.989949553 podStartE2EDuration="1.989949553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:45.986327582 +0000 UTC m=+2238.188578016" watchObservedRunningTime="2026-10-31 01:05:45.989949553 +0000 UTC m=+2238.192199989" Oct 31 01:05:46.317120 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:46.317079 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:46.318619 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:46.318587 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:05:46.318755 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:46.318640 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:05:47.316295 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:47.316260 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:05:47.316470 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:47.316313 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:05:48.316315 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:48.316276 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:05:48.316714 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:48.316334 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:05:49.316296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:49.316258 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:05:49.316481 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:49.316340 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:05:50.316616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:50.316572 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:05:50.317108 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:50.316638 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:05:51.316245 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:51.316199 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:05:51.316433 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:51.316274 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:05:52.317117 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:52.317075 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:05:52.317525 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:52.317127 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:05:53.321450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:53.321409 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:53.321854 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:53.321815 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:05:53.322504 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:05:53.322482 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:06:07.019120 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:06:07.019080 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:06:37.019161 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:06:37.019071 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:06:41.056897 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:41.056853 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr"] Oct 31 01:06:45.139297 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.139251 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr"] Oct 31 01:06:45.144303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.144277 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237284 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237249 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237464 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237303 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzqq\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-kube-api-access-pzzqq\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237464 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237354 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237558 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237501 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237613 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237553 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237655 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237625 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237703 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237655 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237703 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237678 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.237869 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.237843 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.338885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.338847 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339355 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339536 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339438 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339536 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339481 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339670 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339566 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339670 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339571 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339670 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339660 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339726 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzqq\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-kube-api-access-pzzqq\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339758 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.339986 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.339868 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.340119 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.340099 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.341014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.340620 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.341334 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.341277 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.341825 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.341803 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.343376 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.343344 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.344827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.343810 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr"] Oct 31 01:06:45.344827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.343820 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.401297 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.401221 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.401427 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.401303 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzqq\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-kube-api-access-pzzqq\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.465844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.465806 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:45.893943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.893913 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr"] Oct 31 01:06:45.894636 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:06:45.894609 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5d6be7_ff57_4a2c_a7d4_2b298679c55e.slice/crio-94e99a094674d7b88be06939692ff9e818d9093295a8a5d4fbfcaae08f64b298 WatchSource:0}: Error finding container 94e99a094674d7b88be06939692ff9e818d9093295a8a5d4fbfcaae08f64b298: Status 404 returned error can't find the container with id 94e99a094674d7b88be06939692ff9e818d9093295a8a5d4fbfcaae08f64b298 Oct 31 01:06:45.897069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.897036 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:06:45.897158 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.897105 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:06:45.897158 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:45.897139 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:06:46.058075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.057903 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" podUID="cebe79fe-5fa3-49c5-a490-f640f57fe999" containerName="payload-pre-processing" containerID="cri-o://cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8" gracePeriod=25 Oct 31 01:06:46.325813 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.325750 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" event={"ID":"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e","Type":"ContainerStarted","Data":"46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150"} Oct 31 01:06:46.326210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.325842 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" event={"ID":"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e","Type":"ContainerStarted","Data":"94e99a094674d7b88be06939692ff9e818d9093295a8a5d4fbfcaae08f64b298"} Oct 31 01:06:46.447282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.447047 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podStartSLOduration=1.447027353 podStartE2EDuration="1.447027353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:46.442046437 +0000 UTC m=+2298.644296869" watchObservedRunningTime="2026-10-31 01:06:46.447027353 +0000 UTC m=+2298.649277790" Oct 31 01:06:46.466422 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.466396 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:46.468099 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.468070 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 01:06:46.468234 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.468125 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 01:06:46.609389 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:46.609003 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" podUID="cebe79fe-5fa3-49c5-a490-f640f57fe999" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:06:47.467807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:47.467305 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 01:06:47.467807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:47.467374 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 01:06:48.470810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:48.466247 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 01:06:48.470810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:48.466315 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 01:06:49.467808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:49.466936 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 01:06:49.467808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:49.466995 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 01:06:49.983879 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:49.983843 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l"] Oct 31 01:06:50.475326 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:50.475283 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 01:06:50.475546 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:50.475344 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 01:06:51.467810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:51.467196 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 01:06:51.467810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:51.467262 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 01:06:52.467035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:52.467002 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 01:06:52.467218 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:52.467057 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 01:06:53.587810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:53.587181 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:53.591805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:53.589132 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:53.591805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:53.590878 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:06:54.764470 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:54.764426 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n"] Oct 31 01:06:54.764994 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:54.764690 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" podUID="fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" containerName="maas-api" containerID="cri-o://fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108" gracePeriod=30 Oct 31 01:06:54.994839 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:54.985311 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" podUID="3f75dd97-30d3-495c-ab40-6930d43163ff" containerName="payload-pre-processing" containerID="cri-o://b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879" gracePeriod=25 Oct 31 01:06:55.134015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.133990 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:06:55.251179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.251147 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls\") pod \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " Oct 31 01:06:55.251353 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.251190 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-metrics-certs\") pod \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " Oct 31 01:06:55.251451 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.251435 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-openshift-service-ca\") pod \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " Oct 31 01:06:55.251510 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.251496 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-kube-api-access-w8nf7\" (UniqueName: \"kubernetes.io/projected/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-kube-api-access-w8nf7\") pod \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\" (UID: \"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0\") " Oct 31 01:06:55.251962 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.251932 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-openshift-service-ca" pod "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" (UID: "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:55.255524 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.255247 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls" pod "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" (UID: "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:55.262963 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.257224 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-metrics-certs" pod "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" (UID: "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:55.262963 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.262113 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-kube-api-access-w8nf7" pod "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" (UID: "fdee4d51-fbf2-416f-ad9d-f57e651a4ff0"). InnerVolumeSpecName "kube-api-access-w8nf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:55.291994 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.291964 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:06:55.352895 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.352865 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f75dd97-30d3-495c-ab40-6930d43163ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f75dd97-30d3-495c-ab40-6930d43163ff-plugins-config-volume\") pod \"3f75dd97-30d3-495c-ab40-6930d43163ff\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " Oct 31 01:06:55.353063 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.353041 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f75dd97-30d3-495c-ab40-6930d43163ff-kube-api-access-nhngr\" (UniqueName: \"kubernetes.io/projected/3f75dd97-30d3-495c-ab40-6930d43163ff-kube-api-access-nhngr\") pod \"3f75dd97-30d3-495c-ab40-6930d43163ff\" (UID: \"3f75dd97-30d3-495c-ab40-6930d43163ff\") " Oct 31 01:06:55.353251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.353226 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f75dd97-30d3-495c-ab40-6930d43163ff-plugins-config-volume" pod "3f75dd97-30d3-495c-ab40-6930d43163ff" (UID: "3f75dd97-30d3-495c-ab40-6930d43163ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:55.353402 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.353390 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8nf7\" (UniqueName: \"kubernetes.io/projected/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-kube-api-access-w8nf7\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.353443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.353406 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.353443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.353415 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.353443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.353424 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f75dd97-30d3-495c-ab40-6930d43163ff-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.353443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.353435 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.355094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.355068 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f75dd97-30d3-495c-ab40-6930d43163ff-kube-api-access-nhngr" pod "3f75dd97-30d3-495c-ab40-6930d43163ff" (UID: "3f75dd97-30d3-495c-ab40-6930d43163ff"). InnerVolumeSpecName "kube-api-access-nhngr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:55.446397 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.446356 5890 generic.go:356] "Generic (PLEG): container finished" podID="3f75dd97-30d3-495c-ab40-6930d43163ff" containerID="b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879" exitCode=0 Oct 31 01:06:55.446397 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.446396 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" Oct 31 01:06:55.446654 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.446433 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" event={"ID":"3f75dd97-30d3-495c-ab40-6930d43163ff","Type":"ContainerDied","Data":"b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879"} Oct 31 01:06:55.446654 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.446491 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l" event={"ID":"3f75dd97-30d3-495c-ab40-6930d43163ff","Type":"ContainerDied","Data":"ebb79561b6283727a4ae213b692b976e9069da14ea35f3033227e8360026fa46"} Oct 31 01:06:55.446654 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.446520 5890 scope.go:122] "RemoveContainer" containerID="b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879" Oct 31 01:06:55.447989 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.447965 5890 generic.go:356] "Generic (PLEG): container finished" podID="fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" containerID="fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108" exitCode=0 Oct 31 01:06:55.448119 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.448008 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" Oct 31 01:06:55.448238 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.448218 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" event={"ID":"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0","Type":"ContainerDied","Data":"fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108"} Oct 31 01:06:55.448365 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.448342 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n" event={"ID":"fdee4d51-fbf2-416f-ad9d-f57e651a4ff0","Type":"ContainerDied","Data":"d5b7e977af758a09273c1b94acdce28e1989f8a16d3dffbdadb53159af870a8f"} Oct 31 01:06:55.455179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.454383 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhngr\" (UniqueName: \"kubernetes.io/projected/3f75dd97-30d3-495c-ab40-6930d43163ff-kube-api-access-nhngr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:55.458503 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.458463 5890 scope.go:122] "RemoveContainer" containerID="b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879" Oct 31 01:06:55.458748 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:06:55.458725 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879\": container with ID starting with b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879 not found: ID does not exist" containerID="b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879" Oct 31 01:06:55.458859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.458752 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879"} err="failed to get container status \"b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879\": rpc error: code = NotFound desc = could not find container \"b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879\": container with ID starting with b52ae19da22a2e828ad50500b941e67c8d3fa5682ae2129ef8a947ad1f4e5879 not found: ID does not exist" Oct 31 01:06:55.458859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.458770 5890 scope.go:122] "RemoveContainer" containerID="fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108" Oct 31 01:06:55.468351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.468330 5890 scope.go:122] "RemoveContainer" containerID="fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108" Oct 31 01:06:55.468645 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:06:55.468624 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108\": container with ID starting with fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108 not found: ID does not exist" containerID="fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108" Oct 31 01:06:55.468748 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.468654 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108"} err="failed to get container status \"fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108\": rpc error: code = NotFound desc = could not find container \"fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108\": container with ID starting with fc49bcca0dcadd2862d1c1bdce73d801049d35703494b921932431c13ee1e108 not found: ID does not exist" Oct 31 01:06:55.606301 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.606214 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n"] Oct 31 01:06:55.614288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.614261 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31ae40-5fb86fdb96-rdj2n"] Oct 31 01:06:55.755807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.755433 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l"] Oct 31 01:06:55.778722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.778672 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4ab4aa-69d999c6c5-kdx2l"] Oct 31 01:06:55.937935 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.937898 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f75dd97-30d3-495c-ab40-6930d43163ff" path="/var/lib/kubelet/pods/3f75dd97-30d3-495c-ab40-6930d43163ff/volumes" Oct 31 01:06:55.938577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:55.938553 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" path="/var/lib/kubelet/pods/fdee4d51-fbf2-416f-ad9d-f57e651a4ff0/volumes" Oct 31 01:06:56.605239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:56.605201 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" podUID="cebe79fe-5fa3-49c5-a490-f640f57fe999" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:06:59.988936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:59.988905 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l"] Oct 31 01:06:59.989436 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:06:59.989135 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" containerID="cri-o://9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c" gracePeriod=30 Oct 31 01:07:00.036207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:00.036171 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k"] Oct 31 01:07:00.036446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:00.036419 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" containerID="cri-o://b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87" gracePeriod=30 Oct 31 01:07:00.979145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:00.979101 5890 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-pqv5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:07:00.979302 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:00.979157 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:07:02.359392 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:02.359355 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd"] Oct 31 01:07:02.359912 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:02.359565 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" containerID="cri-o://a495e02547d6e9f9367022a59906813082eb8d6bfa899413936b22cbbf486894" gracePeriod=30 Oct 31 01:07:05.332804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.332758 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 01:07:05.366135 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.366107 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 01:07:05.382985 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.382951 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-envoy\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383154 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383003 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-kube-api-access-kpmkm\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-kube-api-access-kpmkm\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383154 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383034 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4e48e6-2013-426f-b584-10c4f04e0034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4e48e6-2013-426f-b584-10c4f04e0034-istiod-ca-cert\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383154 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383134 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-data\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383308 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383230 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-certs\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383308 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383278 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-socket\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383413 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383359 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-token\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-token\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383413 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383390 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-credential-socket\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383416 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-data" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.383540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383480 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-podinfo\") pod \"3d4e48e6-2013-426f-b584-10c4f04e0034\" (UID: \"3d4e48e6-2013-426f-b584-10c4f04e0034\") " Oct 31 01:07:05.383540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383494 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4e48e6-2013-426f-b584-10c4f04e0034-istiod-ca-cert" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:05.383702 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383611 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-socket" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.383843 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.383771 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-credential-socket" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.384205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.384119 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4e48e6-2013-426f-b584-10c4f04e0034-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.384205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.384148 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.384205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.384164 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.384205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.384180 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.384475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.384217 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-certs" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.385416 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.385395 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-envoy" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.385493 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.385415 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-kube-api-access-kpmkm" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "kube-api-access-kpmkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:05.385567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.385549 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-podinfo" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:07:05.385839 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.385825 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-token" pod "3d4e48e6-2013-426f-b584-10c4f04e0034" (UID: "3d4e48e6-2013-426f-b584-10c4f04e0034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:05.484577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484544 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-socket\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.484758 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484601 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-envoy\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.484758 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484688 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-certs\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.484758 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484717 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-istio-token\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-istio-token\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.484758 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484753 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedae545-0e22-4793-a720-230efee85446-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedae545-0e22-4793-a720-230efee85446-istiod-ca-cert\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.485023 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484780 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-data\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.485023 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484857 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-socket" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.485023 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484906 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aedae545-0e22-4793-a720-230efee85446-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedae545-0e22-4793-a720-230efee85446-istio-podinfo\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.485023 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.484979 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-kube-api-access-xnx9s\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-kube-api-access-xnx9s\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.485230 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.485041 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-credential-socket\") pod \"aedae545-0e22-4793-a720-230efee85446\" (UID: \"aedae545-0e22-4793-a720-230efee85446\") " Oct 31 01:07:05.485289 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.485248 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-data" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.485390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.485367 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedae545-0e22-4793-a720-230efee85446-istiod-ca-cert" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:05.485532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.485511 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-credential-socket" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.485644 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.485622 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-certs" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.486354 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486328 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486360 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpmkm\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-kube-api-access-kpmkm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486375 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486387 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4e48e6-2013-426f-b584-10c4f04e0034-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486402 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486415 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedae545-0e22-4793-a720-230efee85446-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486429 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486441 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486453 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4e48e6-2013-426f-b584-10c4f04e0034-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.486469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.486467 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.487561 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.487534 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-istio-token" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:05.488156 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.488118 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-envoy" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.488513 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.488486 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aedae545-0e22-4793-a720-230efee85446-istio-podinfo" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:07:05.489429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.489402 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-kube-api-access-xnx9s" pod "aedae545-0e22-4793-a720-230efee85446" (UID: "aedae545-0e22-4793-a720-230efee85446"). InnerVolumeSpecName "kube-api-access-xnx9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:05.542628 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.542590 5890 generic.go:356] "Generic (PLEG): container finished" podID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerID="b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87" exitCode=0 Oct 31 01:07:05.542849 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.542743 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" event={"ID":"3d4e48e6-2013-426f-b584-10c4f04e0034","Type":"ContainerDied","Data":"b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87"} Oct 31 01:07:05.542913 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.542850 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" event={"ID":"3d4e48e6-2013-426f-b584-10c4f04e0034","Type":"ContainerDied","Data":"0e9c2b76985f493837e345d3bc3a54cd18e3c2273b9a5504bd14306c0d698176"} Oct 31 01:07:05.542913 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.542880 5890 scope.go:122] "RemoveContainer" containerID="b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87" Oct 31 01:07:05.543026 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.543012 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k" Oct 31 01:07:05.546712 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.546681 5890 generic.go:356] "Generic (PLEG): container finished" podID="aedae545-0e22-4793-a720-230efee85446" containerID="9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c" exitCode=0 Oct 31 01:07:05.546901 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.546880 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" event={"ID":"aedae545-0e22-4793-a720-230efee85446","Type":"ContainerDied","Data":"9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c"} Oct 31 01:07:05.546946 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.546934 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" event={"ID":"aedae545-0e22-4793-a720-230efee85446","Type":"ContainerDied","Data":"6e113b755dadfd1038211d9e31a429635f2e3b4ef827c6a30436e95b950ab0c6"} Oct 31 01:07:05.547041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.547027 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l" Oct 31 01:07:05.557794 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.557755 5890 scope.go:122] "RemoveContainer" containerID="b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87" Oct 31 01:07:05.558150 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:07:05.558122 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87\": container with ID starting with b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87 not found: ID does not exist" containerID="b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87" Oct 31 01:07:05.558236 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.558163 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87"} err="failed to get container status \"b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87\": rpc error: code = NotFound desc = could not find container \"b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87\": container with ID starting with b45353187d958c03d35346e062a0d40664cfe6b032844ae4fcf0fb12c2ab2f87 not found: ID does not exist" Oct 31 01:07:05.558236 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.558189 5890 scope.go:122] "RemoveContainer" containerID="9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c" Oct 31 01:07:05.568991 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.568973 5890 scope.go:122] "RemoveContainer" containerID="7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff" Oct 31 01:07:05.578797 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.578765 5890 scope.go:122] "RemoveContainer" containerID="9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c" Oct 31 01:07:05.579090 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:07:05.579051 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c\": container with ID starting with 9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c not found: ID does not exist" containerID="9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c" Oct 31 01:07:05.579170 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.579091 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c"} err="failed to get container status \"9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c\": rpc error: code = NotFound desc = could not find container \"9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c\": container with ID starting with 9abf6dff185571ac410677593421708b26ca4ada8eca13111d221676c205f18c not found: ID does not exist" Oct 31 01:07:05.579170 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.579112 5890 scope.go:122] "RemoveContainer" containerID="7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff" Oct 31 01:07:05.579348 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:07:05.579332 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff\": container with ID starting with 7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff not found: ID does not exist" containerID="7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff" Oct 31 01:07:05.579401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.579351 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff"} err="failed to get container status \"7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff\": rpc error: code = NotFound desc = could not find container \"7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff\": container with ID starting with 7fafe3f06f9593fe5203172f5a059e5f5b619518bd5121fda7f2d11e1a1ba7ff not found: ID does not exist" Oct 31 01:07:05.588414 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.588384 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedae545-0e22-4793-a720-230efee85446-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.588414 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.588413 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnx9s\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-kube-api-access-xnx9s\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.588550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.588425 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedae545-0e22-4793-a720-230efee85446-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.588550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.588438 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedae545-0e22-4793-a720-230efee85446-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.729918 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.729886 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l"] Oct 31 01:07:05.739832 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.736448 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-pqv5l"] Oct 31 01:07:05.842829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.842712 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k"] Oct 31 01:07:05.851146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.851116 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-b687k"] Oct 31 01:07:05.933093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.933058 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" path="/var/lib/kubelet/pods/3d4e48e6-2013-426f-b584-10c4f04e0034/volumes" Oct 31 01:07:05.933565 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:05.933551 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedae545-0e22-4793-a720-230efee85446" path="/var/lib/kubelet/pods/aedae545-0e22-4793-a720-230efee85446/volumes" Oct 31 01:07:06.358302 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.358275 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 01:07:06.398150 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.398119 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cebe79fe-5fa3-49c5-a490-f640f57fe999-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cebe79fe-5fa3-49c5-a490-f640f57fe999-plugins-config-volume\") pod \"cebe79fe-5fa3-49c5-a490-f640f57fe999\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " Oct 31 01:07:06.398322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.398230 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cebe79fe-5fa3-49c5-a490-f640f57fe999-kube-api-access-xpvlj\" (UniqueName: \"kubernetes.io/projected/cebe79fe-5fa3-49c5-a490-f640f57fe999-kube-api-access-xpvlj\") pod \"cebe79fe-5fa3-49c5-a490-f640f57fe999\" (UID: \"cebe79fe-5fa3-49c5-a490-f640f57fe999\") " Oct 31 01:07:06.398560 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.398531 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebe79fe-5fa3-49c5-a490-f640f57fe999-plugins-config-volume" pod "cebe79fe-5fa3-49c5-a490-f640f57fe999" (UID: "cebe79fe-5fa3-49c5-a490-f640f57fe999"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:06.398772 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.398755 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cebe79fe-5fa3-49c5-a490-f640f57fe999-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:06.400575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.400547 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebe79fe-5fa3-49c5-a490-f640f57fe999-kube-api-access-xpvlj" pod "cebe79fe-5fa3-49c5-a490-f640f57fe999" (UID: "cebe79fe-5fa3-49c5-a490-f640f57fe999"). InnerVolumeSpecName "kube-api-access-xpvlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:06.500119 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.499907 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpvlj\" (UniqueName: \"kubernetes.io/projected/cebe79fe-5fa3-49c5-a490-f640f57fe999-kube-api-access-xpvlj\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:06.557656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.557622 5890 generic.go:356] "Generic (PLEG): container finished" podID="cebe79fe-5fa3-49c5-a490-f640f57fe999" containerID="cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8" exitCode=0 Oct 31 01:07:06.557873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.557670 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" Oct 31 01:07:06.557873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.557713 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" event={"ID":"cebe79fe-5fa3-49c5-a490-f640f57fe999","Type":"ContainerDied","Data":"cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8"} Oct 31 01:07:06.557873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.557772 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr" event={"ID":"cebe79fe-5fa3-49c5-a490-f640f57fe999","Type":"ContainerDied","Data":"3b19f34f6f54e567ebd9555bb3c38c0e0108acc8d80ec44ec1121650e66a57de"} Oct 31 01:07:06.557873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.557814 5890 scope.go:122] "RemoveContainer" containerID="cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8" Oct 31 01:07:06.568975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.568953 5890 scope.go:122] "RemoveContainer" containerID="cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8" Oct 31 01:07:06.569290 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:07:06.569266 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8\": container with ID starting with cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8 not found: ID does not exist" containerID="cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8" Oct 31 01:07:06.569344 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.569301 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8"} err="failed to get container status \"cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8\": rpc error: code = NotFound desc = could not find container \"cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8\": container with ID starting with cdaa790d2447b6b5887ed80663958772b1e8258dbffcdbaf27831944c0b94db8 not found: ID does not exist" Oct 31 01:07:06.716713 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.716659 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr"] Oct 31 01:07:06.720332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:06.720307 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c08e7-699cfc9bd7-gglmr"] Oct 31 01:07:07.020043 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:07:07.020001 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:07:07.580635 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.580598 5890 generic.go:356] "Generic (PLEG): container finished" podID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerID="a495e02547d6e9f9367022a59906813082eb8d6bfa899413936b22cbbf486894" exitCode=0 Oct 31 01:07:07.581149 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.580814 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" event={"ID":"e5c2e460-a5c1-4aa7-adce-52feeb05781c","Type":"ContainerDied","Data":"a495e02547d6e9f9367022a59906813082eb8d6bfa899413936b22cbbf486894"} Oct 31 01:07:07.647896 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.647861 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 01:07:07.713980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.713942 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-token\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714164 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714102 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-data\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714164 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714151 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-socket\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714298 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714241 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-certs\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714353 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714301 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istiod-ca-cert\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714353 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714345 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-credential-socket\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714552 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714389 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-podinfo\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714552 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714488 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-envoy\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714552 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714526 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-kube-api-access-whksz\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-kube-api-access-whksz\") pod \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\" (UID: \"e5c2e460-a5c1-4aa7-adce-52feeb05781c\") " Oct 31 01:07:07.714718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714482 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-data" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:07.714718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.714698 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-credential-socket" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:07.715035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.715008 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-certs" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:07.715223 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.715163 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.715223 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.715188 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.715223 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.715203 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.715452 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.715350 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-socket" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:07.715895 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.715868 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istiod-ca-cert" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:07.717354 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.717326 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-kube-api-access-whksz" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "kube-api-access-whksz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:07.717974 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.717953 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-envoy" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:07.718068 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.717978 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-podinfo" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:07:07.718356 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.718335 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-token" pod "e5c2e460-a5c1-4aa7-adce-52feeb05781c" (UID: "e5c2e460-a5c1-4aa7-adce-52feeb05781c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:07.817805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.817391 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.817805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.817428 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.817805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.817484 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.817805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.817499 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.817805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.817513 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whksz\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-kube-api-access-whksz\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.817805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.817530 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2e460-a5c1-4aa7-adce-52feeb05781c-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.935631 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:07.935586 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cebe79fe-5fa3-49c5-a490-f640f57fe999" path="/var/lib/kubelet/pods/cebe79fe-5fa3-49c5-a490-f640f57fe999/volumes" Oct 31 01:07:08.589054 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:08.589021 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" Oct 31 01:07:08.589544 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:08.589052 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd" event={"ID":"e5c2e460-a5c1-4aa7-adce-52feeb05781c","Type":"ContainerDied","Data":"60ee9d69649a59e9a3e7dbc1854541936fa3b3b7f04579f37ef17556c71a9cf8"} Oct 31 01:07:08.589544 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:08.589119 5890 scope.go:122] "RemoveContainer" containerID="a495e02547d6e9f9367022a59906813082eb8d6bfa899413936b22cbbf486894" Oct 31 01:07:08.797040 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:08.797006 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd"] Oct 31 01:07:08.844700 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:08.844621 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-54tvd"] Oct 31 01:07:09.934526 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:09.934491 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" path="/var/lib/kubelet/pods/e5c2e460-a5c1-4aa7-adce-52feeb05781c/volumes" Oct 31 01:07:15.148223 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148179 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs"] Oct 31 01:07:15.148779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148763 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3f75dd97-30d3-495c-ab40-6930d43163ff" containerName="payload-pre-processing" Oct 31 01:07:15.148779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148796 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f75dd97-30d3-495c-ab40-6930d43163ff" containerName="payload-pre-processing" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148812 5890 cpu_manager.go:414] "containerMap: removing container" podUID="fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" containerName="maas-api" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148817 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" containerName="maas-api" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148831 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148836 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148844 5890 cpu_manager.go:414] "containerMap: removing container" podUID="cebe79fe-5fa3-49c5-a490-f640f57fe999" containerName="payload-pre-processing" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148849 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cebe79fe-5fa3-49c5-a490-f640f57fe999" containerName="payload-pre-processing" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148876 5890 cpu_manager.go:414] "containerMap: removing container" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148881 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148895 5890 cpu_manager.go:414] "containerMap: removing container" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148899 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148910 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" Oct 31 01:07:15.148992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.148915 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" Oct 31 01:07:15.149623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.149006 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" Oct 31 01:07:15.149623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.149017 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedae545-0e22-4793-a720-230efee85446" containerName="istio-proxy" Oct 31 01:07:15.149623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.149025 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4e48e6-2013-426f-b584-10c4f04e0034" containerName="istio-proxy" Oct 31 01:07:15.149623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.149033 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f75dd97-30d3-495c-ab40-6930d43163ff" containerName="payload-pre-processing" Oct 31 01:07:15.149623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.149039 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdee4d51-fbf2-416f-ad9d-f57e651a4ff0" containerName="maas-api" Oct 31 01:07:15.149623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.149046 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c2e460-a5c1-4aa7-adce-52feeb05781c" containerName="istio-proxy" Oct 31 01:07:15.149623 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.149051 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="cebe79fe-5fa3-49c5-a490-f640f57fe999" containerName="payload-pre-processing" Oct 31 01:07:15.153941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.153893 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.211730 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.211691 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f5f024-0884-40ea-af34-956365138271-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.211921 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.211837 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.212003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.211976 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f5f024-0884-40ea-af34-956365138271-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.212066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.212019 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfwt\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-kube-api-access-6qfwt\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.212066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.212055 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.212151 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.212127 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.212241 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.212221 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.212296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.212281 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.212340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.212316 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.259796 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.259746 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs"] Oct 31 01:07:15.313172 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313114 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f5f024-0884-40ea-af34-956365138271-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313172 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313171 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfwt\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-kube-api-access-6qfwt\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313437 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313203 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313437 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313226 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313437 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313269 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313437 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313319 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313437 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313349 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313474 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f5f024-0884-40ea-af34-956365138271-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313553 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313725 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313885 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.313941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313927 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.314039 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.314000 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f5f024-0884-40ea-af34-956365138271-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.314039 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.313994 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.315584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.315561 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.316167 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.316145 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f5f024-0884-40ea-af34-956365138271-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.372483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.372448 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.372669 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.372570 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfwt\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-kube-api-access-6qfwt\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.468270 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.468235 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:15.891931 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.891834 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs"] Oct 31 01:07:15.924092 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:07:15.924059 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f5f024_0884_40ea_af34_956365138271.slice/crio-894acb9f73f5935f0b079fc92565cdb400c8de0d4077856f4ea9a2b6e78d1a3b WatchSource:0}: Error finding container 894acb9f73f5935f0b079fc92565cdb400c8de0d4077856f4ea9a2b6e78d1a3b: Status 404 returned error can't find the container with id 894acb9f73f5935f0b079fc92565cdb400c8de0d4077856f4ea9a2b6e78d1a3b Oct 31 01:07:15.926675 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.926630 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:07:15.926758 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.926720 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:07:15.926808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:15.926763 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:07:16.644387 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:16.644345 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" event={"ID":"63f5f024-0884-40ea-af34-956365138271","Type":"ContainerStarted","Data":"f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5"} Oct 31 01:07:16.644951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:16.644410 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" event={"ID":"63f5f024-0884-40ea-af34-956365138271","Type":"ContainerStarted","Data":"894acb9f73f5935f0b079fc92565cdb400c8de0d4077856f4ea9a2b6e78d1a3b"} Oct 31 01:07:16.810961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:16.810906 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podStartSLOduration=1.81089221 podStartE2EDuration="1.81089221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:16.808905057 +0000 UTC m=+2329.011155490" watchObservedRunningTime="2026-10-31 01:07:16.81089221 +0000 UTC m=+2329.013142640" Oct 31 01:07:17.473807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:17.473037 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:17.477806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:17.474511 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 01:07:17.477806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:17.474571 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 01:07:18.477806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:18.474082 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 01:07:18.477806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:18.474154 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 01:07:19.474303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:19.474264 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 01:07:19.475057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:19.475023 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 01:07:20.472469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:20.472431 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 01:07:20.473131 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:20.473096 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 01:07:21.469213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:21.469172 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 01:07:21.469490 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:21.469465 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 01:07:22.469295 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:22.469258 5890 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 01:07:22.469952 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:22.469870 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 01:07:23.136169 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.136129 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c"] Oct 31 01:07:23.141629 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.141598 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.192374 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.192331 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3f25cd-openshift-default-dockercfg-p2kt7\"" Oct 31 01:07:23.268674 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.268608 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c"] Oct 31 01:07:23.281772 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.281735 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-credential-socket\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.282138 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.282117 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-token\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.282332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.282317 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-podinfo\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.282511 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.282496 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-certs\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.282693 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.282678 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-envoy\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.282982 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.282964 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-socket\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.283200 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.283184 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-data\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.283377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.283362 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422778d0-57d5-4d23-8e71-8e60c8ed36be-istiod-ca-cert\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.283559 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.283544 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvdf\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-kube-api-access-hhvdf\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.384996 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-socket\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385245 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385063 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-data\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385245 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385098 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422778d0-57d5-4d23-8e71-8e60c8ed36be-istiod-ca-cert\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385245 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385152 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvdf\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-kube-api-access-hhvdf\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385260 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-credential-socket\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385311 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-token\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385343 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-podinfo\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385381 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-certs\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385624 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385429 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-envoy\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385624 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385543 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-data\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.385908 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.385885 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-socket\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.386076 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.386024 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422778d0-57d5-4d23-8e71-8e60c8ed36be-istiod-ca-cert\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.386525 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.386461 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-credential-socket\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.386742 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.386719 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-certs\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.389419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.389124 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-podinfo\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.389419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.389364 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-envoy\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.445294 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.445255 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-token\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.445459 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.445342 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvdf\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-kube-api-access-hhvdf\") pod \"e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.464257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.463199 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:23.648507 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.647196 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:23.648507 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.647970 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:23.649506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.649473 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:07:23.923598 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.923565 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c"] Oct 31 01:07:23.935388 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.935340 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:07:23.935500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.935437 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:07:23.935500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:23.935483 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:07:24.805299 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:24.805187 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" event={"ID":"422778d0-57d5-4d23-8e71-8e60c8ed36be","Type":"ContainerStarted","Data":"8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3"} Oct 31 01:07:24.805299 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:24.805263 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" event={"ID":"422778d0-57d5-4d23-8e71-8e60c8ed36be","Type":"ContainerStarted","Data":"53ada47f100ac7e18eed82dc90d736ded17f8a106fec8da2f421dc53b9c23201"} Oct 31 01:07:24.926968 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:24.926912 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" podStartSLOduration=1.926897466 podStartE2EDuration="1.926897466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:24.924638105 +0000 UTC m=+2337.126888539" watchObservedRunningTime="2026-10-31 01:07:24.926897466 +0000 UTC m=+2337.129147899" Oct 31 01:07:25.469809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:25.465849 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:25.469809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:25.466598 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 01:07:25.469809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:25.466672 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 01:07:26.468810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:26.468186 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 01:07:26.468810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:26.468250 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 01:07:27.471844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:27.465154 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 01:07:27.471844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:27.465214 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 01:07:28.464231 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:28.464189 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 01:07:28.464527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:28.464493 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 01:07:29.465266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:29.465173 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 01:07:29.465876 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:29.465844 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 01:07:30.468701 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:30.468663 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:30.469192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:30.469058 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:30.469934 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:07:30.469910 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:07:37.018258 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:07:37.018215 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:08:07.018730 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:08:07.018688 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:08:14.155140 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:14.155084 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7"] Oct 31 01:08:18.395671 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.395631 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp"] Oct 31 01:08:18.396089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.395923 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" podUID="3bf6f33a-7b69-40d0-add0-2320348ca472" containerName="maas-api" containerID="cri-o://7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15" gracePeriod=30 Oct 31 01:08:18.703284 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.703253 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:08:18.863178 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.863122 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls\") pod \"3bf6f33a-7b69-40d0-add0-2320348ca472\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " Oct 31 01:08:18.863465 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.863444 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-metrics-certs\") pod \"3bf6f33a-7b69-40d0-add0-2320348ca472\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " Oct 31 01:08:18.863605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.863590 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf6f33a-7b69-40d0-add0-2320348ca472-kube-api-access-8nz7p\" (UniqueName: \"kubernetes.io/projected/3bf6f33a-7b69-40d0-add0-2320348ca472-kube-api-access-8nz7p\") pod \"3bf6f33a-7b69-40d0-add0-2320348ca472\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " Oct 31 01:08:18.863996 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.863979 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf6f33a-7b69-40d0-add0-2320348ca472-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6f33a-7b69-40d0-add0-2320348ca472-openshift-service-ca\") pod \"3bf6f33a-7b69-40d0-add0-2320348ca472\" (UID: \"3bf6f33a-7b69-40d0-add0-2320348ca472\") " Oct 31 01:08:18.864906 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.864875 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf6f33a-7b69-40d0-add0-2320348ca472-openshift-service-ca" pod "3bf6f33a-7b69-40d0-add0-2320348ca472" (UID: "3bf6f33a-7b69-40d0-add0-2320348ca472"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:18.868168 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.868134 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-metrics-certs" pod "3bf6f33a-7b69-40d0-add0-2320348ca472" (UID: "3bf6f33a-7b69-40d0-add0-2320348ca472"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:18.869704 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.869678 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf6f33a-7b69-40d0-add0-2320348ca472-kube-api-access-8nz7p" pod "3bf6f33a-7b69-40d0-add0-2320348ca472" (UID: "3bf6f33a-7b69-40d0-add0-2320348ca472"). InnerVolumeSpecName "kube-api-access-8nz7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:18.873007 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.872985 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls" pod "3bf6f33a-7b69-40d0-add0-2320348ca472" (UID: "3bf6f33a-7b69-40d0-add0-2320348ca472"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:18.966336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.966241 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf6f33a-7b69-40d0-add0-2320348ca472-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:18.966336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.966275 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:18.966336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.966294 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf6f33a-7b69-40d0-add0-2320348ca472-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:18.966336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:18.966307 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nz7p\" (UniqueName: \"kubernetes.io/projected/3bf6f33a-7b69-40d0-add0-2320348ca472-kube-api-access-8nz7p\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:19.155581 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.155527 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" podUID="653bfba6-e4c6-43bb-995d-41c9be1c2033" containerName="payload-pre-processing" containerID="cri-o://54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7" gracePeriod=25 Oct 31 01:08:19.271499 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.271414 5890 generic.go:356] "Generic (PLEG): container finished" podID="3bf6f33a-7b69-40d0-add0-2320348ca472" containerID="7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15" exitCode=0 Oct 31 01:08:19.271499 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.271454 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" event={"ID":"3bf6f33a-7b69-40d0-add0-2320348ca472","Type":"ContainerDied","Data":"7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15"} Oct 31 01:08:19.271738 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.271466 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" Oct 31 01:08:19.271738 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.271527 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp" event={"ID":"3bf6f33a-7b69-40d0-add0-2320348ca472","Type":"ContainerDied","Data":"731201cadfe6c893da02047c89851d4d0fa00320ec675390902a7a1aef942c04"} Oct 31 01:08:19.271738 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.271557 5890 scope.go:122] "RemoveContainer" containerID="7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15" Oct 31 01:08:19.283703 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.283678 5890 scope.go:122] "RemoveContainer" containerID="7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15" Oct 31 01:08:19.284051 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:08:19.284027 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15\": container with ID starting with 7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15 not found: ID does not exist" containerID="7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15" Oct 31 01:08:19.284114 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.284060 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15"} err="failed to get container status \"7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15\": rpc error: code = NotFound desc = could not find container \"7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15\": container with ID starting with 7bdc613bff16b0423fdf8a3940d94d8b9479300d166456438d37787620ac4e15 not found: ID does not exist" Oct 31 01:08:19.437753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.437695 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp"] Oct 31 01:08:19.444352 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.444286 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c08e7-c49d9b87f-vqvjp"] Oct 31 01:08:19.508504 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.508478 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:08:19.682880 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.682854 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653bfba6-e4c6-43bb-995d-41c9be1c2033-kube-api-access-ckqns\" (UniqueName: \"kubernetes.io/projected/653bfba6-e4c6-43bb-995d-41c9be1c2033-kube-api-access-ckqns\") pod \"653bfba6-e4c6-43bb-995d-41c9be1c2033\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " Oct 31 01:08:19.683061 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.682990 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653bfba6-e4c6-43bb-995d-41c9be1c2033-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653bfba6-e4c6-43bb-995d-41c9be1c2033-plugins-config-volume\") pod \"653bfba6-e4c6-43bb-995d-41c9be1c2033\" (UID: \"653bfba6-e4c6-43bb-995d-41c9be1c2033\") " Oct 31 01:08:19.683359 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.683334 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653bfba6-e4c6-43bb-995d-41c9be1c2033-plugins-config-volume" pod "653bfba6-e4c6-43bb-995d-41c9be1c2033" (UID: "653bfba6-e4c6-43bb-995d-41c9be1c2033"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:19.685084 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.685062 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653bfba6-e4c6-43bb-995d-41c9be1c2033-kube-api-access-ckqns" pod "653bfba6-e4c6-43bb-995d-41c9be1c2033" (UID: "653bfba6-e4c6-43bb-995d-41c9be1c2033"). InnerVolumeSpecName "kube-api-access-ckqns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:19.785093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.785055 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckqns\" (UniqueName: \"kubernetes.io/projected/653bfba6-e4c6-43bb-995d-41c9be1c2033-kube-api-access-ckqns\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:19.785093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.785096 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653bfba6-e4c6-43bb-995d-41c9be1c2033-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:19.933875 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:19.933771 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf6f33a-7b69-40d0-add0-2320348ca472" path="/var/lib/kubelet/pods/3bf6f33a-7b69-40d0-add0-2320348ca472/volumes" Oct 31 01:08:20.280457 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.280366 5890 generic.go:356] "Generic (PLEG): container finished" podID="653bfba6-e4c6-43bb-995d-41c9be1c2033" containerID="54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7" exitCode=0 Oct 31 01:08:20.280457 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.280403 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" event={"ID":"653bfba6-e4c6-43bb-995d-41c9be1c2033","Type":"ContainerDied","Data":"54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7"} Oct 31 01:08:20.280457 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.280436 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" Oct 31 01:08:20.280457 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.280448 5890 scope.go:122] "RemoveContainer" containerID="54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7" Oct 31 01:08:20.280766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.280439 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7" event={"ID":"653bfba6-e4c6-43bb-995d-41c9be1c2033","Type":"ContainerDied","Data":"c4a06c09fdaab5163f7a2eebe40ed4cbd4a14931b5d695b385fdb95f005e0c31"} Oct 31 01:08:20.290145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.290128 5890 scope.go:122] "RemoveContainer" containerID="54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7" Oct 31 01:08:20.290457 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:08:20.290419 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7\": container with ID starting with 54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7 not found: ID does not exist" containerID="54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7" Oct 31 01:08:20.290565 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.290464 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7"} err="failed to get container status \"54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7\": rpc error: code = NotFound desc = could not find container \"54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7\": container with ID starting with 54c0823567a20111caf71839794b028f8229167515582b291d411147f50226f7 not found: ID does not exist" Oct 31 01:08:20.440322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.440276 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7"] Oct 31 01:08:20.445358 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:20.445332 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-add5bf-898b54d47-blfp7"] Oct 31 01:08:21.934638 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:21.934600 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653bfba6-e4c6-43bb-995d-41c9be1c2033" path="/var/lib/kubelet/pods/653bfba6-e4c6-43bb-995d-41c9be1c2033/volumes" Oct 31 01:08:28.830241 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:28.830212 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:08:28.943639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:28.943610 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:08:37.024507 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:08:37.024467 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:08:38.659584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:38.659545 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph"] Oct 31 01:08:38.660011 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:38.659856 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" containerID="cri-o://877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c" gracePeriod=30 Oct 31 01:08:42.880739 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:42.880692 5890 patch_prober.go:29] interesting pod/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:08:42.881256 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:42.880774 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:08:43.937875 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:43.937851 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:08:44.087731 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.087696 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-istio-token\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-istio-token\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089011 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-credential-socket\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089203 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089088 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecbca718-dec3-44fc-a165-6c372866230b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbca718-dec3-44fc-a165-6c372866230b-istiod-ca-cert\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089203 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089147 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecbca718-dec3-44fc-a165-6c372866230b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbca718-dec3-44fc-a165-6c372866230b-istio-podinfo\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089341 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089233 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-envoy\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089341 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089302 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-certs\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089461 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089339 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-data\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089461 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089378 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-kube-api-access-dph5w\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-kube-api-access-dph5w\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.089461 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.089408 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-socket\") pod \"ecbca718-dec3-44fc-a165-6c372866230b\" (UID: \"ecbca718-dec3-44fc-a165-6c372866230b\") " Oct 31 01:08:44.090916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.090889 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbca718-dec3-44fc-a165-6c372866230b-istiod-ca-cert" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:44.091457 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.091290 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-certs" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.091457 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.091420 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-credential-socket" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.091707 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.091685 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-istio-token" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:44.091873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.091713 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-data" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.092353 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.092142 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-socket" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.093749 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.093718 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-envoy" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.094322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.094290 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-kube-api-access-dph5w" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "kube-api-access-dph5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:44.094428 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.094325 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecbca718-dec3-44fc-a165-6c372866230b-istio-podinfo" pod "ecbca718-dec3-44fc-a165-6c372866230b" (UID: "ecbca718-dec3-44fc-a165-6c372866230b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:08:44.191271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191194 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191227 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191241 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbca718-dec3-44fc-a165-6c372866230b-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191254 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbca718-dec3-44fc-a165-6c372866230b-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191265 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191274 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191283 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191293 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dph5w\" (UniqueName: \"kubernetes.io/projected/ecbca718-dec3-44fc-a165-6c372866230b-kube-api-access-dph5w\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.191577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.191303 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbca718-dec3-44fc-a165-6c372866230b-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.453438 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.453356 5890 generic.go:356] "Generic (PLEG): container finished" podID="ecbca718-dec3-44fc-a165-6c372866230b" containerID="877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c" exitCode=0 Oct 31 01:08:44.453438 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.453401 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" event={"ID":"ecbca718-dec3-44fc-a165-6c372866230b","Type":"ContainerDied","Data":"877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c"} Oct 31 01:08:44.453438 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.453425 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" Oct 31 01:08:44.453676 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.453447 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph" event={"ID":"ecbca718-dec3-44fc-a165-6c372866230b","Type":"ContainerDied","Data":"03b6e8dcf7fa96e50a1f301427504575a170f686f8751a44675bddcd95b95625"} Oct 31 01:08:44.453676 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.453469 5890 scope.go:122] "RemoveContainer" containerID="877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c" Oct 31 01:08:44.465296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.465277 5890 scope.go:122] "RemoveContainer" containerID="7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a" Oct 31 01:08:44.476697 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.476633 5890 scope.go:122] "RemoveContainer" containerID="877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c" Oct 31 01:08:44.476985 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:08:44.476945 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c\": container with ID starting with 877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c not found: ID does not exist" containerID="877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c" Oct 31 01:08:44.477108 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.476992 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c"} err="failed to get container status \"877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c\": rpc error: code = NotFound desc = could not find container \"877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c\": container with ID starting with 877d64b55cef3676fa3889d10cf99ee56db1e6d5f8cabf2800c8291bcd1ea28c not found: ID does not exist" Oct 31 01:08:44.477108 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.477011 5890 scope.go:122] "RemoveContainer" containerID="7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a" Oct 31 01:08:44.477304 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:08:44.477288 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a\": container with ID starting with 7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a not found: ID does not exist" containerID="7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a" Oct 31 01:08:44.477361 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.477309 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a"} err="failed to get container status \"7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a\": rpc error: code = NotFound desc = could not find container \"7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a\": container with ID starting with 7e44f7f99b88a68352add879a459cc9149fbfcaa43c5569b0d53c7afa910942a not found: ID does not exist" Oct 31 01:08:44.614349 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.614311 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph"] Oct 31 01:08:44.662067 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:44.662033 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-add5bf-openshift-default-cc4d5db49-gr8ph"] Oct 31 01:08:45.934505 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:08:45.934472 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbca718-dec3-44fc-a165-6c372866230b" path="/var/lib/kubelet/pods/ecbca718-dec3-44fc-a165-6c372866230b/volumes" Oct 31 01:09:07.018478 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:09:07.018432 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:09:13.620617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:13.620535 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp"] Oct 31 01:09:13.621028 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:13.620751 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" containerID="cri-o://4d5388655222875c5f0c66350ea3473c4b5c838c8fbdca1933c4a0638e170c92" gracePeriod=30 Oct 31 01:09:18.409550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.409495 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn"] Oct 31 01:09:18.410312 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410288 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3bf6f33a-7b69-40d0-add0-2320348ca472" containerName="maas-api" Oct 31 01:09:18.410429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410312 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf6f33a-7b69-40d0-add0-2320348ca472" containerName="maas-api" Oct 31 01:09:18.410429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410335 5890 cpu_manager.go:414] "containerMap: removing container" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" Oct 31 01:09:18.410429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410342 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" Oct 31 01:09:18.410429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410352 5890 cpu_manager.go:414] "containerMap: removing container" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" Oct 31 01:09:18.410429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410361 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" Oct 31 01:09:18.410429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410373 5890 cpu_manager.go:414] "containerMap: removing container" podUID="653bfba6-e4c6-43bb-995d-41c9be1c2033" containerName="payload-pre-processing" Oct 31 01:09:18.410429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410380 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653bfba6-e4c6-43bb-995d-41c9be1c2033" containerName="payload-pre-processing" Oct 31 01:09:18.410816 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410556 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf6f33a-7b69-40d0-add0-2320348ca472" containerName="maas-api" Oct 31 01:09:18.410816 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410572 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" Oct 31 01:09:18.410816 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410581 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbca718-dec3-44fc-a165-6c372866230b" containerName="istio-proxy" Oct 31 01:09:18.410816 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.410591 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="653bfba6-e4c6-43bb-995d-41c9be1c2033" containerName="payload-pre-processing" Oct 31 01:09:18.414502 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.414478 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.444957 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.444903 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.445095 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.445012 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.445168 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.445083 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.445872 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.445835 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.446011 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.445928 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.446011 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.445954 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pszr\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-kube-api-access-9pszr\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.446094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.446062 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.446128 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.446094 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.446128 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.446121 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d4ef84-2c51-4a65-8a07-409da5190c45-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.467069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.467018 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ef2572-openshift-default-dockercfg-fvb72\"" Oct 31 01:09:18.519173 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.519137 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn"] Oct 31 01:09:18.547026 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.546977 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.547322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.547288 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pszr\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-kube-api-access-9pszr\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.547554 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.547525 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.547707 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.547691 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.547867 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.547850 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d4ef84-2c51-4a65-8a07-409da5190c45-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.548107 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.548076 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.548242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.547849 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.548679 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.548657 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.549185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.548830 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d4ef84-2c51-4a65-8a07-409da5190c45-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.549333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.549089 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.549464 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.549447 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.549602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.549574 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.550104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.550068 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.550272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.549463 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.550919 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.550893 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.553081 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.553032 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.619134 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.619098 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.625343 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.625314 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pszr\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-kube-api-access-9pszr\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.729236 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.729200 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:18.782631 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.782569 5890 generic.go:356] "Generic (PLEG): container finished" podID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerID="4d5388655222875c5f0c66350ea3473c4b5c838c8fbdca1933c4a0638e170c92" exitCode=0 Oct 31 01:09:18.782872 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:18.782846 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" event={"ID":"ac77a6db-df07-4cd5-a205-9858cf0be656","Type":"ContainerDied","Data":"4d5388655222875c5f0c66350ea3473c4b5c838c8fbdca1933c4a0638e170c92"} Oct 31 01:09:19.005961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.005882 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 01:09:19.063439 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063376 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac77a6db-df07-4cd5-a205-9858cf0be656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac77a6db-df07-4cd5-a205-9858cf0be656-istiod-ca-cert\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063452 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-envoy\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063508 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-token\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-token\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063606 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-certs\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063639 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-credential-socket\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063712 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-podinfo\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063802 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-data\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063813 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac77a6db-df07-4cd5-a205-9858cf0be656-istiod-ca-cert" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:19.063992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063836 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-socket\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063895 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-kube-api-access-wn5n9\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-kube-api-access-wn5n9\") pod \"ac77a6db-df07-4cd5-a205-9858cf0be656\" (UID: \"ac77a6db-df07-4cd5-a205-9858cf0be656\") " Oct 31 01:09:19.063992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.063956 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-certs" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:19.064207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064029 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-credential-socket" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:19.064207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064168 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-socket" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:19.064318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064283 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-data" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:19.064568 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064469 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.064568 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064500 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.064568 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064519 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac77a6db-df07-4cd5-a205-9858cf0be656-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.064568 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064532 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.064568 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.064547 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.066403 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.066375 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-token" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:19.066512 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.066417 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-kube-api-access-wn5n9" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "kube-api-access-wn5n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:19.066512 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.066467 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-podinfo" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:09:19.066607 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.066592 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-envoy" pod "ac77a6db-df07-4cd5-a205-9858cf0be656" (UID: "ac77a6db-df07-4cd5-a205-9858cf0be656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:19.189294 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.189261 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5n9\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-kube-api-access-wn5n9\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.189294 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.189295 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.189946 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.189309 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.189946 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.189320 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac77a6db-df07-4cd5-a205-9858cf0be656-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.202264 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.199598 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn"] Oct 31 01:09:19.209817 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:09:19.206987 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d4ef84_2c51_4a65_8a07_409da5190c45.slice/crio-be4e4be3f4a9e17982baa664320935cb1e7be15300f3fb295053c6d02bef8ab7 WatchSource:0}: Error finding container be4e4be3f4a9e17982baa664320935cb1e7be15300f3fb295053c6d02bef8ab7: Status 404 returned error can't find the container with id be4e4be3f4a9e17982baa664320935cb1e7be15300f3fb295053c6d02bef8ab7 Oct 31 01:09:19.210338 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.210291 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:19.210428 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.210390 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:19.210489 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.210436 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:19.793072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.793036 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" Oct 31 01:09:19.793072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.793063 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp" event={"ID":"ac77a6db-df07-4cd5-a205-9858cf0be656","Type":"ContainerDied","Data":"a1312e478a727c2cfe8fb314f41d0b9f097efe1ab37aea2a564f83c46b2cfd74"} Oct 31 01:09:19.793638 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.793130 5890 scope.go:122] "RemoveContainer" containerID="4d5388655222875c5f0c66350ea3473c4b5c838c8fbdca1933c4a0638e170c92" Oct 31 01:09:19.795948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.795922 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" event={"ID":"71d4ef84-2c51-4a65-8a07-409da5190c45","Type":"ContainerStarted","Data":"77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4"} Oct 31 01:09:19.796108 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.795965 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" event={"ID":"71d4ef84-2c51-4a65-8a07-409da5190c45","Type":"ContainerStarted","Data":"be4e4be3f4a9e17982baa664320935cb1e7be15300f3fb295053c6d02bef8ab7"} Oct 31 01:09:19.955829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:19.955720 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp"] Oct 31 01:09:20.008366 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:20.008324 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-7b9dp"] Oct 31 01:09:20.123822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:20.122657 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podStartSLOduration=2.122639361 podStartE2EDuration="2.122639361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:20.119941745 +0000 UTC m=+2452.322192177" watchObservedRunningTime="2026-10-31 01:09:20.122639361 +0000 UTC m=+2452.324889796" Oct 31 01:09:20.730104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:20.730064 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:20.731348 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:20.731322 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 01:09:20.731422 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:20.731379 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 01:09:21.729624 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:21.729586 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 01:09:21.730055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:21.729640 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 01:09:21.933440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:21.933401 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" path="/var/lib/kubelet/pods/ac77a6db-df07-4cd5-a205-9858cf0be656/volumes" Oct 31 01:09:22.729592 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:22.729563 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 01:09:22.729753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:22.729608 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 01:09:23.729945 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:23.729912 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 01:09:23.730341 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:23.729973 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 01:09:24.730620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:24.730583 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 01:09:24.731038 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:24.730646 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 01:09:25.730242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:25.730201 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 31 01:09:25.730433 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:25.730263 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 31 01:09:26.856312 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:26.856272 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:26.857382 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:26.857357 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:26.857903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:26.857880 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:09:33.545048 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.545012 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk"] Oct 31 01:09:33.545763 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.545740 5890 cpu_manager.go:414] "containerMap: removing container" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" Oct 31 01:09:33.545899 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.545764 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" Oct 31 01:09:33.546014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.545998 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac77a6db-df07-4cd5-a205-9858cf0be656" containerName="istio-proxy" Oct 31 01:09:33.550740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.550712 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.600883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.600846 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc1db1d-0066-42a5-8316-a66bda6e2408-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601082 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.600905 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601082 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.601018 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601082 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.601045 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.601102 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.601132 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-kube-api-access-nbhz5\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.601242 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.601285 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.601429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.601322 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.651141 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.647861 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk"] Oct 31 01:09:33.702745 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.702712 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.702947 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.702754 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.702952 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc1db1d-0066-42a5-8316-a66bda6e2408-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.702995 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703104 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703123 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703164 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703187 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-kube-api-access-nbhz5\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703370 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703249 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703381 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703659 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703588 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703659 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703627 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703835 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703729 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.703835 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.703759 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc1db1d-0066-42a5-8316-a66bda6e2408-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.705526 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.705500 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.705838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.705822 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.789806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.788812 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.816099 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.816029 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-kube-api-access-nbhz5\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:33.896371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:33.896332 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:34.323317 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:34.323278 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk"] Oct 31 01:09:34.323515 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:09:34.323493 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc1db1d_0066_42a5_8316_a66bda6e2408.slice/crio-fe18307ce185e4fb746ec22045921f5475ed5df0cda0348c04d9c0a791562101 WatchSource:0}: Error finding container fe18307ce185e4fb746ec22045921f5475ed5df0cda0348c04d9c0a791562101: Status 404 returned error can't find the container with id fe18307ce185e4fb746ec22045921f5475ed5df0cda0348c04d9c0a791562101 Oct 31 01:09:34.325975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:34.325938 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:34.326071 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:34.326007 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:34.326071 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:34.326036 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:34.928823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:34.928775 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" event={"ID":"6fc1db1d-0066-42a5-8316-a66bda6e2408","Type":"ContainerStarted","Data":"7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf"} Oct 31 01:09:34.929228 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:34.928836 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" event={"ID":"6fc1db1d-0066-42a5-8316-a66bda6e2408","Type":"ContainerStarted","Data":"fe18307ce185e4fb746ec22045921f5475ed5df0cda0348c04d9c0a791562101"} Oct 31 01:09:35.048265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:35.048206 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podStartSLOduration=2.048191529 podStartE2EDuration="2.048191529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:35.044531369 +0000 UTC m=+2467.246781802" watchObservedRunningTime="2026-10-31 01:09:35.048191529 +0000 UTC m=+2467.250441962" Oct 31 01:09:35.896526 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:35.896480 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:35.898059 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:35.898029 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 01:09:35.898200 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:35.898081 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 01:09:36.897412 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:36.897377 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 01:09:36.897843 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:36.897433 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 01:09:37.020652 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:09:37.020616 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:09:37.287812 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:37.282619 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c"] Oct 31 01:09:37.287812 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:37.282917 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" containerID="cri-o://8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3" gracePeriod=30 Oct 31 01:09:37.897856 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:37.897826 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 01:09:37.898298 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:37.897877 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 01:09:38.897023 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:38.896991 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 01:09:38.897330 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:38.897043 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 01:09:39.897452 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:39.897411 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 01:09:39.897888 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:39.897473 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 01:09:40.897622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:40.897580 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 31 01:09:40.898179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:40.897642 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 31 01:09:42.223135 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.223094 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:42.223620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.223597 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:42.224346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.224315 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:09:42.610516 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.610479 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:09:42.721572 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721541 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-envoy\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.721776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721607 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-podinfo\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.721776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721634 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-token\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-token\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.721776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721651 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-data\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.721776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721684 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-socket\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.721776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721799 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-certs\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.722184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721854 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-kube-api-access-hhvdf\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-kube-api-access-hhvdf\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.722184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.721886 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422778d0-57d5-4d23-8e71-8e60c8ed36be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422778d0-57d5-4d23-8e71-8e60c8ed36be-istiod-ca-cert\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.722184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.722090 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-credential-socket\") pod \"422778d0-57d5-4d23-8e71-8e60c8ed36be\" (UID: \"422778d0-57d5-4d23-8e71-8e60c8ed36be\") " Oct 31 01:09:42.722349 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.722319 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-data" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:42.722859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.722631 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.722859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.722706 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-socket" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:42.723038 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.722977 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-certs" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:42.723279 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.723162 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-credential-socket" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:42.723622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.723601 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422778d0-57d5-4d23-8e71-8e60c8ed36be-istiod-ca-cert" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:42.724320 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.724294 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-token" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:42.724419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.724389 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-envoy" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:42.724582 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.724565 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-kube-api-access-hhvdf" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "kube-api-access-hhvdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:42.724992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.724976 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-podinfo" pod "422778d0-57d5-4d23-8e71-8e60c8ed36be" (UID: "422778d0-57d5-4d23-8e71-8e60c8ed36be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:09:42.823480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823393 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.823480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823426 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.823480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823440 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.823480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823451 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.823480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823464 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.823480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823478 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422778d0-57d5-4d23-8e71-8e60c8ed36be-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.823859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823491 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhvdf\" (UniqueName: \"kubernetes.io/projected/422778d0-57d5-4d23-8e71-8e60c8ed36be-kube-api-access-hhvdf\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:42.823859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:42.823505 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422778d0-57d5-4d23-8e71-8e60c8ed36be-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:43.081722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.081624 5890 generic.go:356] "Generic (PLEG): container finished" podID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerID="8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3" exitCode=0 Oct 31 01:09:43.083291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.082977 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" event={"ID":"422778d0-57d5-4d23-8e71-8e60c8ed36be","Type":"ContainerDied","Data":"8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3"} Oct 31 01:09:43.083291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.083104 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" Oct 31 01:09:43.083291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.083119 5890 scope.go:122] "RemoveContainer" containerID="8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3" Oct 31 01:09:43.085057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.083102 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c" event={"ID":"422778d0-57d5-4d23-8e71-8e60c8ed36be","Type":"ContainerDied","Data":"53ada47f100ac7e18eed82dc90d736ded17f8a106fec8da2f421dc53b9c23201"} Oct 31 01:09:43.108042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.108015 5890 scope.go:122] "RemoveContainer" containerID="8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3" Oct 31 01:09:43.108403 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:09:43.108379 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3\": container with ID starting with 8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3 not found: ID does not exist" containerID="8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3" Oct 31 01:09:43.108481 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.108416 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3"} err="failed to get container status \"8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3\": rpc error: code = NotFound desc = could not find container \"8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3\": container with ID starting with 8e8538ac18b099d0f463234cd5e031cfe86a4e02e971bfb4a086a391a45325f3 not found: ID does not exist" Oct 31 01:09:43.253576 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.253535 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c"] Oct 31 01:09:43.295986 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.295951 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3f25cd-openshift-default-576b49df5c-fnh7c"] Oct 31 01:09:43.933549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:43.933513 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" path="/var/lib/kubelet/pods/422778d0-57d5-4d23-8e71-8e60c8ed36be/volumes" Oct 31 01:09:51.223188 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.223151 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw"] Oct 31 01:09:51.223677 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.223386 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" podUID="02152a8e-0532-46e2-802e-6e711b205881" containerName="maas-api" containerID="cri-o://c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168" gracePeriod=30 Oct 31 01:09:51.504735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.504705 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2"] Oct 31 01:09:51.505257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.505243 5890 cpu_manager.go:414] "containerMap: removing container" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" Oct 31 01:09:51.505306 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.505258 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" Oct 31 01:09:51.505393 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.505383 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="422778d0-57d5-4d23-8e71-8e60c8ed36be" containerName="istio-proxy" Oct 31 01:09:51.509252 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.509228 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.515881 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.515862 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:09:51.560025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.559986 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-66e263-openshift-default-dockercfg-7sjc5\"" Oct 31 01:09:51.567629 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.567599 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2"] Oct 31 01:09:51.638751 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.638716 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-metrics-certs\") pod \"02152a8e-0532-46e2-802e-6e711b205881\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " Oct 31 01:09:51.639051 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.639027 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02152a8e-0532-46e2-802e-6e711b205881-kube-api-access-n5rzr\" (UniqueName: \"kubernetes.io/projected/02152a8e-0532-46e2-802e-6e711b205881-kube-api-access-n5rzr\") pod \"02152a8e-0532-46e2-802e-6e711b205881\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " Oct 31 01:09:51.639188 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.639173 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02152a8e-0532-46e2-802e-6e711b205881-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02152a8e-0532-46e2-802e-6e711b205881-openshift-service-ca\") pod \"02152a8e-0532-46e2-802e-6e711b205881\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " Oct 31 01:09:51.639329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.639316 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls\") pod \"02152a8e-0532-46e2-802e-6e711b205881\" (UID: \"02152a8e-0532-46e2-802e-6e711b205881\") " Oct 31 01:09:51.639984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.639957 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-data\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.640152 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.640134 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-socket\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.640269 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.640252 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjbv\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-kube-api-access-krjbv\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.640421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.640407 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-token\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.640552 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.640538 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80169d-7e18-47d8-b7df-37f303823dcf-istiod-ca-cert\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.640774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.640758 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-credential-socket\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.640968 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.640950 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-podinfo\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.641092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.641079 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-certs\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.641219 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.641207 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-envoy\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.641822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.641799 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02152a8e-0532-46e2-802e-6e711b205881-openshift-service-ca" pod "02152a8e-0532-46e2-802e-6e711b205881" (UID: "02152a8e-0532-46e2-802e-6e711b205881"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:51.642137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.641943 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02152a8e-0532-46e2-802e-6e711b205881-kube-api-access-n5rzr" pod "02152a8e-0532-46e2-802e-6e711b205881" (UID: "02152a8e-0532-46e2-802e-6e711b205881"). InnerVolumeSpecName "kube-api-access-n5rzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:51.646827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.644248 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls" pod "02152a8e-0532-46e2-802e-6e711b205881" (UID: "02152a8e-0532-46e2-802e-6e711b205881"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:51.646827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.644342 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-metrics-certs" pod "02152a8e-0532-46e2-802e-6e711b205881" (UID: "02152a8e-0532-46e2-802e-6e711b205881"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:51.742598 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742463 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-token\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.742598 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742554 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80169d-7e18-47d8-b7df-37f303823dcf-istiod-ca-cert\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742674 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-credential-socket\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742725 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-podinfo\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742766 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-certs\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742850 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-envoy\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742948 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-data\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.742983 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-socket\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743011 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjbv\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-kube-api-access-krjbv\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743106 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.743550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743123 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rzr\" (UniqueName: \"kubernetes.io/projected/02152a8e-0532-46e2-802e-6e711b205881-kube-api-access-n5rzr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.743550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743139 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02152a8e-0532-46e2-802e-6e711b205881-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.743550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743154 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02152a8e-0532-46e2-802e-6e711b205881-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.743550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743188 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-credential-socket\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743511 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80169d-7e18-47d8-b7df-37f303823dcf-istiod-ca-cert\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743668 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-data\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743706 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-socket\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.743918 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.743841 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-certs\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.745981 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.745957 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-envoy\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.746409 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.746387 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-podinfo\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.824713 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.824667 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjbv\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-kube-api-access-krjbv\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:51.831068 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:51.830481 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-token\") pod \"e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:52.125001 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.124908 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:52.160533 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.160501 5890 generic.go:356] "Generic (PLEG): container finished" podID="02152a8e-0532-46e2-802e-6e711b205881" containerID="c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168" exitCode=0 Oct 31 01:09:52.160715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.160561 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" Oct 31 01:09:52.160715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.160587 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" event={"ID":"02152a8e-0532-46e2-802e-6e711b205881","Type":"ContainerDied","Data":"c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168"} Oct 31 01:09:52.160715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.160641 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw" event={"ID":"02152a8e-0532-46e2-802e-6e711b205881","Type":"ContainerDied","Data":"1fab49da4dca5d51391c327e45ead6c8420931126bff0bd59e76a40eefe909ea"} Oct 31 01:09:52.160715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.160670 5890 scope.go:122] "RemoveContainer" containerID="c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168" Oct 31 01:09:52.173372 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.173341 5890 scope.go:122] "RemoveContainer" containerID="c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168" Oct 31 01:09:52.173692 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:09:52.173665 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168\": container with ID starting with c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168 not found: ID does not exist" containerID="c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168" Oct 31 01:09:52.173854 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.173700 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168"} err="failed to get container status \"c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168\": rpc error: code = NotFound desc = could not find container \"c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168\": container with ID starting with c220d90529514056df380240d4960556e162a7ad002528c75108cbb8f5c6f168 not found: ID does not exist" Oct 31 01:09:52.328637 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.328565 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw"] Oct 31 01:09:52.333474 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.333449 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e4801-7dc75458cb-76nvw"] Oct 31 01:09:52.550858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.550827 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2"] Oct 31 01:09:52.552106 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:09:52.551883 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c80169d_7e18_47d8_b7df_37f303823dcf.slice/crio-d84456f3d50080fca4a851f6cf5a988498c6a701bdccf9c3f816eb26634d79ac WatchSource:0}: Error finding container d84456f3d50080fca4a851f6cf5a988498c6a701bdccf9c3f816eb26634d79ac: Status 404 returned error can't find the container with id d84456f3d50080fca4a851f6cf5a988498c6a701bdccf9c3f816eb26634d79ac Oct 31 01:09:52.554622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.554583 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:52.554710 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.554669 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:52.554753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:52.554713 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:09:53.170273 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:53.170240 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" event={"ID":"5c80169d-7e18-47d8-b7df-37f303823dcf","Type":"ContainerStarted","Data":"3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65"} Oct 31 01:09:53.170441 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:53.170291 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" event={"ID":"5c80169d-7e18-47d8-b7df-37f303823dcf","Type":"ContainerStarted","Data":"d84456f3d50080fca4a851f6cf5a988498c6a701bdccf9c3f816eb26634d79ac"} Oct 31 01:09:53.287695 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:53.287641 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" podStartSLOduration=2.287629524 podStartE2EDuration="2.287629524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:53.28700835 +0000 UTC m=+2485.489258783" watchObservedRunningTime="2026-10-31 01:09:53.287629524 +0000 UTC m=+2485.489879957" Oct 31 01:09:53.934089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:53.934050 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02152a8e-0532-46e2-802e-6e711b205881" path="/var/lib/kubelet/pods/02152a8e-0532-46e2-802e-6e711b205881/volumes" Oct 31 01:09:54.125661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:54.125628 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:54.133755 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:54.133714 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:54.179098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:54.179057 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:54.179889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:54.179866 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:09:58.153930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.153900 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt"] Oct 31 01:09:58.154557 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.154540 5890 cpu_manager.go:414] "containerMap: removing container" podUID="02152a8e-0532-46e2-802e-6e711b205881" containerName="maas-api" Oct 31 01:09:58.154557 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.154557 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02152a8e-0532-46e2-802e-6e711b205881" containerName="maas-api" Oct 31 01:09:58.155058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.154653 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="02152a8e-0532-46e2-802e-6e711b205881" containerName="maas-api" Oct 31 01:09:58.157888 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.157873 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.219822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.219772 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.220022 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.219914 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4c13ff-9583-4f2e-909e-549e1286ea06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.220114 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.220093 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwnj\" (UniqueName: \"kubernetes.io/projected/be4c13ff-9583-4f2e-909e-549e1286ea06-kube-api-access-qtwnj\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.220176 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.220147 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.257722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.257695 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-8fb7f0-serving-cert\"" Oct 31 01:09:58.260677 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.260655 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt"] Oct 31 01:09:58.321170 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.321131 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.321346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.321194 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4c13ff-9583-4f2e-909e-549e1286ea06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.321346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.321329 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwnj\" (UniqueName: \"kubernetes.io/projected/be4c13ff-9583-4f2e-909e-549e1286ea06-kube-api-access-qtwnj\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.321467 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.321375 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.321541 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:09:58.321511 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-serving-cert: secret "maas-api-e2e-shared-a-w4-8fb7f0-serving-cert" not found Oct 31 01:09:58.321604 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:09:58.321576 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls podName:be4c13ff-9583-4f2e-909e-549e1286ea06 nodeName:}" failed. No retries permitted until 2026-10-31 01:09:58.821555648 +0000 UTC m=+2491.023806076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls") pod "maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" (UID: "be4c13ff-9583-4f2e-909e-549e1286ea06") : secret "maas-api-e2e-shared-a-w4-8fb7f0-serving-cert" not found Oct 31 01:09:58.322027 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.321997 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4c13ff-9583-4f2e-909e-549e1286ea06-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.323868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.323840 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.383059 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.383026 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwnj\" (UniqueName: \"kubernetes.io/projected/be4c13ff-9583-4f2e-909e-549e1286ea06-kube-api-access-qtwnj\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.829171 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.829140 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:58.831655 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:58.831621 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:59.071663 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:59.071616 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:09:59.458765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:59.458736 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt"] Oct 31 01:09:59.459429 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:09:59.459405 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4c13ff_9583_4f2e_909e_549e1286ea06.slice/crio-f738616a34fcdcbfc5326cbc81d4e61b6c93723196fcf4f73a25fb4d7d59515d WatchSource:0}: Error finding container f738616a34fcdcbfc5326cbc81d4e61b6c93723196fcf4f73a25fb4d7d59515d: Status 404 returned error can't find the container with id f738616a34fcdcbfc5326cbc81d4e61b6c93723196fcf4f73a25fb4d7d59515d Oct 31 01:09:59.460821 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:09:59.460772 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:10:00.222424 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:00.222385 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" event={"ID":"be4c13ff-9583-4f2e-909e-549e1286ea06","Type":"ContainerStarted","Data":"f738616a34fcdcbfc5326cbc81d4e61b6c93723196fcf4f73a25fb4d7d59515d"} Oct 31 01:10:01.254808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:01.249884 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" event={"ID":"be4c13ff-9583-4f2e-909e-549e1286ea06","Type":"ContainerStarted","Data":"331e718dda6ef34947c81d961baf3b33031535d6c1fb4e2e5d4dbdf21a227613"} Oct 31 01:10:01.254808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:01.252624 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:10:01.397204 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:01.397137 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" podStartSLOduration=2.341392109 podStartE2EDuration="3.397118513s" totalImagesPullingTime="1.055726404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:01.394078478 +0000 UTC m=+2493.596328912" watchObservedRunningTime="2026-10-31 01:10:01.397118513 +0000 UTC m=+2493.599368982" Oct 31 01:10:01.541622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:01.541523 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp"] Oct 31 01:10:01.541805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:01.541750 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" containerID="cri-o://7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e" gracePeriod=30 Oct 31 01:10:04.012697 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.012657 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd"] Oct 31 01:10:04.019012 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.018989 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.060051 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.060012 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd"] Oct 31 01:10:04.102551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.102512 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.103156 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.103111 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxcq\" (UniqueName: \"kubernetes.io/projected/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-kube-api-access-qgxcq\") pod \"payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.103523 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.103486 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-66e263\"" Oct 31 01:10:04.103523 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.103501 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-66e263-dockercfg-xfz4n\"" Oct 31 01:10:04.204941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.204907 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.205161 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.205054 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxcq\" (UniqueName: \"kubernetes.io/projected/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-kube-api-access-qgxcq\") pod \"payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.205699 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.205670 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.264681 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.264588 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxcq\" (UniqueName: \"kubernetes.io/projected/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-kube-api-access-qgxcq\") pod \"payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.334762 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.334724 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:04.711436 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:04.711404 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd"] Oct 31 01:10:04.711878 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:10:04.711842 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5eb4f5f_0a4a_47dc_a329_2ef99d44aed1.slice/crio-df9250a8f7bd2eab228f81c302b4df21dbba681b44f366d0d49b9a674a4a3376 WatchSource:0}: Error finding container df9250a8f7bd2eab228f81c302b4df21dbba681b44f366d0d49b9a674a4a3376: Status 404 returned error can't find the container with id df9250a8f7bd2eab228f81c302b4df21dbba681b44f366d0d49b9a674a4a3376 Oct 31 01:10:05.280452 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:05.280417 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" event={"ID":"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1","Type":"ContainerStarted","Data":"55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63"} Oct 31 01:10:05.280862 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:05.280478 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" event={"ID":"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1","Type":"ContainerStarted","Data":"df9250a8f7bd2eab228f81c302b4df21dbba681b44f366d0d49b9a674a4a3376"} Oct 31 01:10:05.280862 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:05.280616 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:05.394843 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:05.394766 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" podStartSLOduration=2.394751067 podStartE2EDuration="2.394751067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:05.393073302 +0000 UTC m=+2497.595323736" watchObservedRunningTime="2026-10-31 01:10:05.394751067 +0000 UTC m=+2497.597001500" Oct 31 01:10:05.865290 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:05.865251 5890 patch_prober.go:29] interesting pod/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:10:05.865465 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:05.865315 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:10:06.820291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.820235 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:10:06.947554 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947517 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-envoy\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.947769 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947642 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-podinfo\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.947769 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947668 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-certs\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.947769 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947705 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-token\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-token\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.947965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947769 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istiod-ca-cert\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.947965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947825 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-credential-socket\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.947965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947863 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-socket\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.948123 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.947971 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-kube-api-access-wd4rw\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-kube-api-access-wd4rw\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.948123 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.948010 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-certs" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:06.948123 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.948040 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-data\") pod \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\" (UID: \"50379ab5-7dc7-4d22-9ea8-194e4be2aba5\") " Oct 31 01:10:06.948276 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.948160 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-credential-socket" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:06.948276 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.948208 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-socket" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:06.950940 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.948372 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-data" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:06.951311 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.951258 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istiod-ca-cert" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:06.952266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.951824 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:06.952266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.951854 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:06.952266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.951871 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:06.952266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.951884 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:06.952266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.951897 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:06.954995 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.954969 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-podinfo" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:10:06.955104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.955015 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-token" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:06.955145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.955112 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-envoy" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:06.955550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:06.955525 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-kube-api-access-wd4rw" pod "50379ab5-7dc7-4d22-9ea8-194e4be2aba5" (UID: "50379ab5-7dc7-4d22-9ea8-194e4be2aba5"). InnerVolumeSpecName "kube-api-access-wd4rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:07.020690 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:10:07.020649 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:10:07.053297 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.053265 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:07.053297 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.053297 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:07.053487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.053313 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4rw\" (UniqueName: \"kubernetes.io/projected/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-kube-api-access-wd4rw\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:07.053487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.053323 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50379ab5-7dc7-4d22-9ea8-194e4be2aba5-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:07.296024 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.295987 5890 generic.go:356] "Generic (PLEG): container finished" podID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerID="7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e" exitCode=0 Oct 31 01:10:07.296235 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.296039 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" Oct 31 01:10:07.296235 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.296076 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" event={"ID":"50379ab5-7dc7-4d22-9ea8-194e4be2aba5","Type":"ContainerDied","Data":"7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e"} Oct 31 01:10:07.296235 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.296135 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp" event={"ID":"50379ab5-7dc7-4d22-9ea8-194e4be2aba5","Type":"ContainerDied","Data":"c6d282032fd95e44885507fb2132335aa88cf69f3a0270a87c6b40859cb0bfcd"} Oct 31 01:10:07.296235 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.296164 5890 scope.go:122] "RemoveContainer" containerID="7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e" Oct 31 01:10:07.307723 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.307704 5890 scope.go:122] "RemoveContainer" containerID="63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657" Oct 31 01:10:07.317051 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.317033 5890 scope.go:122] "RemoveContainer" containerID="7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e" Oct 31 01:10:07.317303 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:10:07.317285 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e\": container with ID starting with 7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e not found: ID does not exist" containerID="7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e" Oct 31 01:10:07.317349 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.317314 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e"} err="failed to get container status \"7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e\": rpc error: code = NotFound desc = could not find container \"7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e\": container with ID starting with 7a971ae0d6e7f47a0026d5fee133f43121f0802da8e06ce132d4eaf43dc8513e not found: ID does not exist" Oct 31 01:10:07.317349 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.317331 5890 scope.go:122] "RemoveContainer" containerID="63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657" Oct 31 01:10:07.317559 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:10:07.317542 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657\": container with ID starting with 63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657 not found: ID does not exist" containerID="63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657" Oct 31 01:10:07.317599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.317566 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657"} err="failed to get container status \"63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657\": rpc error: code = NotFound desc = could not find container \"63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657\": container with ID starting with 63a485296ef0f7ec69c8b09bcfcbc0426fdefee8b6e7526e0adf5636c9ce3657 not found: ID does not exist" Oct 31 01:10:07.455993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.455961 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp"] Oct 31 01:10:07.503947 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.503908 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e4801-openshift-default-f96899fcc-4z4bp"] Oct 31 01:10:07.944447 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:07.944396 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" path="/var/lib/kubelet/pods/50379ab5-7dc7-4d22-9ea8-194e4be2aba5/volumes" Oct 31 01:10:08.269798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:08.269709 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:10:13.777379 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.777337 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84"] Oct 31 01:10:13.777967 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.777950 5890 cpu_manager.go:414] "containerMap: removing container" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" Oct 31 01:10:13.778019 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.777969 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" Oct 31 01:10:13.778019 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.777993 5890 cpu_manager.go:414] "containerMap: removing container" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" Oct 31 01:10:13.778019 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.777998 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" Oct 31 01:10:13.778113 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.778098 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" Oct 31 01:10:13.782991 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.782973 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.842185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.842149 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cb9a07-openshift-default-dockercfg-bmgl6\"" Oct 31 01:10:13.918912 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.918868 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84"] Oct 31 01:10:13.944955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.944918 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-credential-socket\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945114 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.944980 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-podinfo\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.945103 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-token\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.945217 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00e0b1d-36be-4089-96a1-f35924e0fda9-istiod-ca-cert\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.945283 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-envoy\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945437 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.945420 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-certs\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.945495 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-data\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945569 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.945558 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-socket\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:13.945615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:13.945585 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q4sz\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-kube-api-access-9q4sz\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.046969 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-envoy\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047261 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047106 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-certs\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047261 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047149 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-data\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047261 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047197 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-socket\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047261 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047221 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q4sz\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-kube-api-access-9q4sz\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047281 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-credential-socket\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047319 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-podinfo\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047404 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-token\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047475 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00e0b1d-36be-4089-96a1-f35924e0fda9-istiod-ca-cert\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047709 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047640 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-socket\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047709 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047666 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-data\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.047938 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047919 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-certs\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.048006 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.047948 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-credential-socket\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.048260 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.048238 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00e0b1d-36be-4089-96a1-f35924e0fda9-istiod-ca-cert\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.049716 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.049673 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-envoy\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.050169 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.050146 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-podinfo\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.106194 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.106159 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-token\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.106345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.106256 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q4sz\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-kube-api-access-9q4sz\") pod \"e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.397760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.397665 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:14.820991 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.820960 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84"] Oct 31 01:10:14.853761 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:10:14.853732 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00e0b1d_36be_4089_96a1_f35924e0fda9.slice/crio-260f95519c91de163ee1b39e6464e6508dec02bcacba13a701fc6ebf6c918d7a WatchSource:0}: Error finding container 260f95519c91de163ee1b39e6464e6508dec02bcacba13a701fc6ebf6c918d7a: Status 404 returned error can't find the container with id 260f95519c91de163ee1b39e6464e6508dec02bcacba13a701fc6ebf6c918d7a Oct 31 01:10:14.856370 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.856337 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:10:14.856458 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.856403 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:10:14.856458 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:14.856432 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:10:15.350743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:15.350707 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" event={"ID":"b00e0b1d-36be-4089-96a1-f35924e0fda9","Type":"ContainerStarted","Data":"daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00"} Oct 31 01:10:15.350950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:15.350764 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" event={"ID":"b00e0b1d-36be-4089-96a1-f35924e0fda9","Type":"ContainerStarted","Data":"260f95519c91de163ee1b39e6464e6508dec02bcacba13a701fc6ebf6c918d7a"} Oct 31 01:10:15.398261 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:15.398223 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:15.404111 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:15.404087 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:15.474160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:15.474090 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" podStartSLOduration=2.474070895 podStartE2EDuration="2.474070895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:15.469591681 +0000 UTC m=+2507.671842116" watchObservedRunningTime="2026-10-31 01:10:15.474070895 +0000 UTC m=+2507.676321331" Oct 31 01:10:16.291889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:16.291858 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:10:16.359134 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:16.359100 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:16.360079 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:16.360054 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:10:23.631480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.631442 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt"] Oct 31 01:10:23.632370 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.632347 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="50379ab5-7dc7-4d22-9ea8-194e4be2aba5" containerName="istio-proxy" Oct 31 01:10:23.636914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.636892 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:23.682917 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.682888 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc16ba85-0f91-4224-8257-2b27b920a7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:23.683079 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.682964 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6846\" (UniqueName: \"kubernetes.io/projected/dc16ba85-0f91-4224-8257-2b27b920a7d3-kube-api-access-b6846\") pod \"payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:23.691036 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.691005 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt"] Oct 31 01:10:23.742258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.742219 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-cb9a07-dockercfg-6hlhw\"" Oct 31 01:10:23.742653 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.742570 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-cb9a07\"" Oct 31 01:10:23.793810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.792859 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc16ba85-0f91-4224-8257-2b27b920a7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:23.793810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.792982 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6846\" (UniqueName: \"kubernetes.io/projected/dc16ba85-0f91-4224-8257-2b27b920a7d3-kube-api-access-b6846\") pod \"payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:23.798356 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.794387 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc16ba85-0f91-4224-8257-2b27b920a7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:23.855358 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.855322 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6846\" (UniqueName: \"kubernetes.io/projected/dc16ba85-0f91-4224-8257-2b27b920a7d3-kube-api-access-b6846\") pod \"payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:23.952600 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:23.952565 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:24.328511 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:24.328479 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt"] Oct 31 01:10:24.329596 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:10:24.329562 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc16ba85_0f91_4224_8257_2b27b920a7d3.slice/crio-b422bfdd2c93bb46750c5cb32ef5d27f1d41e897ecaf9a4b46a53f211aab1a85 WatchSource:0}: Error finding container b422bfdd2c93bb46750c5cb32ef5d27f1d41e897ecaf9a4b46a53f211aab1a85: Status 404 returned error can't find the container with id b422bfdd2c93bb46750c5cb32ef5d27f1d41e897ecaf9a4b46a53f211aab1a85 Oct 31 01:10:24.418595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:24.418561 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" event={"ID":"dc16ba85-0f91-4224-8257-2b27b920a7d3","Type":"ContainerStarted","Data":"3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473"} Oct 31 01:10:24.418736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:24.418630 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:24.418736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:24.418655 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" event={"ID":"dc16ba85-0f91-4224-8257-2b27b920a7d3","Type":"ContainerStarted","Data":"b422bfdd2c93bb46750c5cb32ef5d27f1d41e897ecaf9a4b46a53f211aab1a85"} Oct 31 01:10:24.531351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:24.531286 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" podStartSLOduration=1.531269495 podStartE2EDuration="1.531269495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:24.530065213 +0000 UTC m=+2516.732315647" watchObservedRunningTime="2026-10-31 01:10:24.531269495 +0000 UTC m=+2516.733519950" Oct 31 01:10:35.429978 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:35.429951 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:10:37.018874 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:10:37.018836 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:10:40.363565 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.363524 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc"] Oct 31 01:10:40.368687 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.368664 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.424186 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.424154 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc"] Oct 31 01:10:40.467183 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.467148 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4397fc-serving-cert\"" Oct 31 01:10:40.507691 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.507640 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.507898 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.507838 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvl9\" (UniqueName: \"kubernetes.io/projected/28e641e9-efa8-4b1d-85e6-53f58407efb8-kube-api-access-2rvl9\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.507898 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.507879 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e641e9-efa8-4b1d-85e6-53f58407efb8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.508068 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.508050 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.609137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.609104 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvl9\" (UniqueName: \"kubernetes.io/projected/28e641e9-efa8-4b1d-85e6-53f58407efb8-kube-api-access-2rvl9\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.609137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.609138 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e641e9-efa8-4b1d-85e6-53f58407efb8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.609364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.609223 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.609364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.609357 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.609934 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.609909 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e641e9-efa8-4b1d-85e6-53f58407efb8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.611840 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.611811 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.612002 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.611979 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.668638 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.668565 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvl9\" (UniqueName: \"kubernetes.io/projected/28e641e9-efa8-4b1d-85e6-53f58407efb8-kube-api-access-2rvl9\") pod \"maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:40.682489 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:40.682461 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:41.067114 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:41.067075 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc"] Oct 31 01:10:41.068898 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:10:41.068865 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e641e9_efa8_4b1d_85e6_53f58407efb8.slice/crio-694f1f32ae610164dbb89d4b7cc3e5d5500f5058ca925bb199bc655083c73d69 WatchSource:0}: Error finding container 694f1f32ae610164dbb89d4b7cc3e5d5500f5058ca925bb199bc655083c73d69: Status 404 returned error can't find the container with id 694f1f32ae610164dbb89d4b7cc3e5d5500f5058ca925bb199bc655083c73d69 Oct 31 01:10:41.554639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:41.554603 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" event={"ID":"28e641e9-efa8-4b1d-85e6-53f58407efb8","Type":"ContainerStarted","Data":"694f1f32ae610164dbb89d4b7cc3e5d5500f5058ca925bb199bc655083c73d69"} Oct 31 01:10:42.566122 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:42.566070 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" event={"ID":"28e641e9-efa8-4b1d-85e6-53f58407efb8","Type":"ContainerStarted","Data":"f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458"} Oct 31 01:10:42.566592 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:42.566149 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:42.681105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:42.681039 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" podStartSLOduration=1.637654683 podStartE2EDuration="2.681019212s" totalImagesPullingTime="1.043364529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:42.679865462 +0000 UTC m=+2534.882115896" watchObservedRunningTime="2026-10-31 01:10:42.681019212 +0000 UTC m=+2534.883269645" Oct 31 01:10:48.580280 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:48.580248 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:10:50.174581 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.174548 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk"] Oct 31 01:10:50.179696 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.179675 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.230737 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.230707 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-66e263-serving-cert\"" Oct 31 01:10:50.233836 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.233804 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-metrics-certs\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.233953 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.233882 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmj75\" (UniqueName: \"kubernetes.io/projected/f4b10aef-764a-431f-8574-b1bb735caaa3-kube-api-access-cmj75\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.234014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.233980 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b10aef-764a-431f-8574-b1bb735caaa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.234059 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.234041 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.283715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.283678 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk"] Oct 31 01:10:50.335170 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.335136 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.335361 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.335238 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-metrics-certs\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.335361 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:10:50.335298 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-serving-cert: secret "maas-api-e2e-ipp-b-66e263-serving-cert" not found Oct 31 01:10:50.335492 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.335305 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmj75\" (UniqueName: \"kubernetes.io/projected/f4b10aef-764a-431f-8574-b1bb735caaa3-kube-api-access-cmj75\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.335492 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:10:50.335368 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls podName:f4b10aef-764a-431f-8574-b1bb735caaa3 nodeName:}" failed. No retries permitted until 2026-10-31 01:10:50.835347634 +0000 UTC m=+2543.037598056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls") pod "maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" (UID: "f4b10aef-764a-431f-8574-b1bb735caaa3") : secret "maas-api-e2e-ipp-b-66e263-serving-cert" not found Oct 31 01:10:50.335492 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.335481 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b10aef-764a-431f-8574-b1bb735caaa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.336249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.336219 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b10aef-764a-431f-8574-b1bb735caaa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.337670 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.337616 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-metrics-certs\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.397436 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.397398 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmj75\" (UniqueName: \"kubernetes.io/projected/f4b10aef-764a-431f-8574-b1bb735caaa3-kube-api-access-cmj75\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.842444 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.842409 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:50.844858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:50.844832 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:51.095591 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:51.095513 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:51.475766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:51.475736 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk"] Oct 31 01:10:51.477072 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:10:51.477045 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b10aef_764a_431f_8574_b1bb735caaa3.slice/crio-3cca388c07cb90f914bf229da2fa5873f3c9c354833560640876969a44ef1f46 WatchSource:0}: Error finding container 3cca388c07cb90f914bf229da2fa5873f3c9c354833560640876969a44ef1f46: Status 404 returned error can't find the container with id 3cca388c07cb90f914bf229da2fa5873f3c9c354833560640876969a44ef1f46 Oct 31 01:10:51.647085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:51.647054 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" event={"ID":"f4b10aef-764a-431f-8574-b1bb735caaa3","Type":"ContainerStarted","Data":"3cca388c07cb90f914bf229da2fa5873f3c9c354833560640876969a44ef1f46"} Oct 31 01:10:52.655485 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:52.655395 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" event={"ID":"f4b10aef-764a-431f-8574-b1bb735caaa3","Type":"ContainerStarted","Data":"5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743"} Oct 31 01:10:52.655485 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:52.655461 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:10:52.772577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:52.772521 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" podStartSLOduration=1.87950477 podStartE2EDuration="2.77250686s" totalImagesPullingTime="893.00209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:52.769569849 +0000 UTC m=+2544.971820281" watchObservedRunningTime="2026-10-31 01:10:52.77250686 +0000 UTC m=+2544.974757289" Oct 31 01:10:58.669398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:10:58.669367 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:11:07.018720 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:11:07.018680 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:11:15.853492 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:15.853454 5890 generic.go:356] "Generic (PLEG): container finished" podID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerID="3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65" exitCode=0 Oct 31 01:11:15.853972 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:15.853525 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" event={"ID":"5c80169d-7e18-47d8-b7df-37f303823dcf","Type":"ContainerDied","Data":"3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65"} Oct 31 01:11:15.853972 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:15.853954 5890 scope.go:122] "RemoveContainer" containerID="3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65" Oct 31 01:11:15.855850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:15.855814 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:15.855967 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:15.855891 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:15.855967 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:15.855926 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:16.879012 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:16.878973 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" event={"ID":"5c80169d-7e18-47d8-b7df-37f303823dcf","Type":"ContainerStarted","Data":"b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e"} Oct 31 01:11:17.126091 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:17.126047 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:11:17.126279 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:17.126241 5890 patch_prober.go:29] interesting pod/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 31 01:11:17.126332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:17.126285 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 31 01:11:18.126069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:18.126031 5890 patch_prober.go:29] interesting pod/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 31 01:11:18.126453 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:18.126092 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 31 01:11:19.125338 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:19.125300 5890 patch_prober.go:29] interesting pod/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 31 01:11:19.125520 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:19.125355 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 31 01:11:20.125284 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:20.125246 5890 patch_prober.go:29] interesting pod/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 31 01:11:20.125875 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:20.125313 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 31 01:11:21.125296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:21.125259 5890 patch_prober.go:29] interesting pod/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 31 01:11:21.125748 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:21.125341 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 31 01:11:22.129943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:22.129909 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:11:22.130446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:22.130283 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:11:22.131073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:22.131053 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:11:30.302932 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.302899 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj"] Oct 31 01:11:30.307831 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.307809 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.366100 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.365205 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-3be6fa-openshift-default-dockercfg-jf82t\"" Oct 31 01:11:30.412909 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.412872 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj"] Oct 31 01:11:30.502093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502057 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-podinfo\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502285 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502115 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istiod-ca-cert\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502285 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502267 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-credential-socket\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502344 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-data\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502397 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-certs\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502466 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502438 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-socket\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502512 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502465 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-token\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502568 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502535 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbgh\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-kube-api-access-bxbgh\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.502659 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.502646 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-envoy\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.603558 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.603474 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istiod-ca-cert\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.603558 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.603531 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-credential-socket\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.603756 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.603576 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-data\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.603756 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.603630 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-certs\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.603756 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.603652 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-socket\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.603756 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.603686 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-token\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.603756 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.603726 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbgh\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-kube-api-access-bxbgh\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.604075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.604009 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-envoy\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.604137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.604116 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-data\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.604197 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.604116 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-podinfo\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.604197 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.604155 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-socket\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.604276 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.604257 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-credential-socket\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.604380 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.604361 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istiod-ca-cert\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.604466 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.604448 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-certs\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.606358 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.606329 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-envoy\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.606567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.606552 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-podinfo\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.663178 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.663137 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-token\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.663411 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.663388 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbgh\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-kube-api-access-bxbgh\") pod \"e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:30.921497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:30.921420 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:31.355632 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:11:31.355603 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9096a8_7cf2_4454_a1c3_73cea55b63fc.slice/crio-584b60d4bf0b4be762f75c4bd543ab1962eb45a6f53073693b077ca4f288dcad WatchSource:0}: Error finding container 584b60d4bf0b4be762f75c4bd543ab1962eb45a6f53073693b077ca4f288dcad: Status 404 returned error can't find the container with id 584b60d4bf0b4be762f75c4bd543ab1962eb45a6f53073693b077ca4f288dcad Oct 31 01:11:31.356080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:31.355832 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj"] Oct 31 01:11:31.358754 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:31.358721 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:31.358869 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:31.358803 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:31.358869 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:31.358838 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:32.018866 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:32.018817 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" event={"ID":"1a9096a8-7cf2-4454-a1c3-73cea55b63fc","Type":"ContainerStarted","Data":"ff65640f1bede847685b9430dc25365b9efea422c10fc9a54b5a99e3536fb5dc"} Oct 31 01:11:32.019061 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:32.018890 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" event={"ID":"1a9096a8-7cf2-4454-a1c3-73cea55b63fc","Type":"ContainerStarted","Data":"584b60d4bf0b4be762f75c4bd543ab1962eb45a6f53073693b077ca4f288dcad"} Oct 31 01:11:32.234933 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:32.234879 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podStartSLOduration=2.234865809 podStartE2EDuration="2.234865809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:32.184856979 +0000 UTC m=+2584.387107412" watchObservedRunningTime="2026-10-31 01:11:32.234865809 +0000 UTC m=+2584.437116302" Oct 31 01:11:32.922330 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:32.922291 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:32.928957 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:32.928931 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:33.026015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:33.025981 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:33.026999 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:33.026973 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:11:37.020752 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:11:37.020706 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:11:43.127864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:43.127736 5890 generic.go:356] "Generic (PLEG): container finished" podID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerID="daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00" exitCode=0 Oct 31 01:11:43.127864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:43.127811 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" event={"ID":"b00e0b1d-36be-4089-96a1-f35924e0fda9","Type":"ContainerDied","Data":"daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00"} Oct 31 01:11:43.128515 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:43.128234 5890 scope.go:122] "RemoveContainer" containerID="daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00" Oct 31 01:11:43.129881 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:43.129853 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:43.129957 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:43.129924 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:43.129996 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:43.129960 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:44.137877 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:44.137834 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" event={"ID":"b00e0b1d-36be-4089-96a1-f35924e0fda9","Type":"ContainerStarted","Data":"26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305"} Oct 31 01:11:44.397943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:44.397852 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:11:44.398892 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:44.398865 5890 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 01:11:44.399013 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:44.398916 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 01:11:45.398409 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:45.398375 5890 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 01:11:45.398836 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:45.398441 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 01:11:46.359913 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:46.359876 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:11:46.398103 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:46.398066 5890 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 01:11:46.398308 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:46.398130 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 01:11:47.398927 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:47.398892 5890 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 01:11:47.399307 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:47.398956 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 01:11:48.398018 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:48.397985 5890 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 01:11:48.398219 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:48.398037 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 01:11:49.402917 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:49.402882 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:11:49.403994 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:11:49.403974 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:12:04.329169 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:04.329129 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd"] Oct 31 01:12:07.019152 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:12:07.019112 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:12:09.330211 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.330124 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" podUID="e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" containerName="payload-processing" containerID="cri-o://55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63" gracePeriod=25 Oct 31 01:12:09.602150 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.602125 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:12:09.670081 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.670051 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-kube-api-access-qgxcq\" (UniqueName: \"kubernetes.io/projected/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-kube-api-access-qgxcq\") pod \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " Oct 31 01:12:09.670249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.670104 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-plugins-config-volume\") pod \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\" (UID: \"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1\") " Oct 31 01:12:09.670607 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.670573 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-plugins-config-volume" pod "e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" (UID: "e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:09.670774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.670755 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:09.672439 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.672407 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-kube-api-access-qgxcq" pod "e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" (UID: "e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1"). InnerVolumeSpecName "kube-api-access-qgxcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:09.771852 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:09.771817 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxcq\" (UniqueName: \"kubernetes.io/projected/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1-kube-api-access-qgxcq\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:12:10.420740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.420645 5890 generic.go:356] "Generic (PLEG): container finished" podID="e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" containerID="55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63" exitCode=0 Oct 31 01:12:10.420740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.420698 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" Oct 31 01:12:10.421337 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.420733 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" event={"ID":"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1","Type":"ContainerDied","Data":"55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63"} Oct 31 01:12:10.421337 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.420819 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd" event={"ID":"e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1","Type":"ContainerDied","Data":"df9250a8f7bd2eab228f81c302b4df21dbba681b44f366d0d49b9a674a4a3376"} Oct 31 01:12:10.421337 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.420851 5890 scope.go:122] "RemoveContainer" containerID="55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63" Oct 31 01:12:10.431143 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.431127 5890 scope.go:122] "RemoveContainer" containerID="55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63" Oct 31 01:12:10.431377 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:12:10.431359 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63\": container with ID starting with 55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63 not found: ID does not exist" containerID="55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63" Oct 31 01:12:10.431450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.431389 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63"} err="failed to get container status \"55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63\": rpc error: code = NotFound desc = could not find container \"55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63\": container with ID starting with 55b8017dc0507c3f5fd6be1d24b91f033f649db585868b23fe871fa3c4fc2e63 not found: ID does not exist" Oct 31 01:12:10.585180 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.585120 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd"] Oct 31 01:12:10.632842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:10.632797 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-f78d47599-bvgvd"] Oct 31 01:12:11.934429 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:11.934392 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" path="/var/lib/kubelet/pods/e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1/volumes" Oct 31 01:12:13.457765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.457733 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk"] Oct 31 01:12:13.458345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.458329 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" containerName="payload-processing" Oct 31 01:12:13.458407 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.458346 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" containerName="payload-processing" Oct 31 01:12:13.458469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.458460 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5eb4f5f-0a4a-47dc-a329-2ef99d44aed1" containerName="payload-processing" Oct 31 01:12:13.461518 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.461500 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.521652 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.521612 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hrf\" (UniqueName: \"kubernetes.io/projected/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-kube-api-access-g4hrf\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.521911 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.521678 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.522256 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.522228 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.522426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.522396 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.561845 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.561765 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a72754-serving-cert\"" Oct 31 01:12:13.565503 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.565470 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk"] Oct 31 01:12:13.623985 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.623945 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hrf\" (UniqueName: \"kubernetes.io/projected/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-kube-api-access-g4hrf\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.624191 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.624003 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.624191 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.624114 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.624346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.624222 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.624405 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:12:13.624367 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-serving-cert: secret "maas-api-e2e-shared-b-w4-a72754-serving-cert" not found Oct 31 01:12:13.624470 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:12:13.624441 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls podName:a6f4edaf-4c2c-4345-9a20-61c27b2fc9af nodeName:}" failed. No retries permitted until 2026-10-31 01:12:14.124419099 +0000 UTC m=+2626.326669529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls") pod "maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" (UID: "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af") : secret "maas-api-e2e-shared-b-w4-a72754-serving-cert" not found Oct 31 01:12:13.624935 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.624909 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.626550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.626524 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:13.684388 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:13.684355 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hrf\" (UniqueName: \"kubernetes.io/projected/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-kube-api-access-g4hrf\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:14.131795 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:14.131759 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:14.134289 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:14.134259 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:14.375595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:14.375560 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:14.753380 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:14.753337 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk"] Oct 31 01:12:14.754545 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:12:14.754515 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f4edaf_4c2c_4345_9a20_61c27b2fc9af.slice/crio-cc0fe472892d9b957231667acd92625cf36160dd620b95646dc33ab058c8b2d8 WatchSource:0}: Error finding container cc0fe472892d9b957231667acd92625cf36160dd620b95646dc33ab058c8b2d8: Status 404 returned error can't find the container with id cc0fe472892d9b957231667acd92625cf36160dd620b95646dc33ab058c8b2d8 Oct 31 01:12:15.462430 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:15.462395 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" event={"ID":"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af","Type":"ContainerStarted","Data":"cc0fe472892d9b957231667acd92625cf36160dd620b95646dc33ab058c8b2d8"} Oct 31 01:12:16.472375 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:16.472323 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" event={"ID":"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af","Type":"ContainerStarted","Data":"2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7"} Oct 31 01:12:16.472927 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:16.472899 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:16.591817 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:16.591722 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" podStartSLOduration=2.714204273 podStartE2EDuration="3.591703552s" totalImagesPullingTime="877.499279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:16.586775905 +0000 UTC m=+2628.789026337" watchObservedRunningTime="2026-10-31 01:12:16.591703552 +0000 UTC m=+2628.793953982" Oct 31 01:12:22.485447 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:22.485416 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:12:32.823805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.823756 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z"] Oct 31 01:12:32.827736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.827708 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.885568 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.885535 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-7f8596-openshift-default-dockercfg-wmfrf\"" Oct 31 01:12:32.909866 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.909822 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-podinfo\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.920844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.910204 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-credential-socket\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.921137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.921110 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-envoy\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.921307 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.921289 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-socket\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.921415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.921399 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-token\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.921530 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.921514 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-data\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.921651 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.921636 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcnbq\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-kube-api-access-rcnbq\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.921856 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.921833 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-certs\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.922201 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.922178 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3815179-7ff3-4db0-b671-ecdafd4e977c-istiod-ca-cert\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:32.946368 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:32.946326 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z"] Oct 31 01:12:33.025717 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.025669 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-podinfo\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.025937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.025773 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-credential-socket\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.025937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.025820 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-envoy\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.025937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.025873 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-socket\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.025937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.025897 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-token\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.026118 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.025942 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-data\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.026118 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.026002 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcnbq\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-kube-api-access-rcnbq\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.026118 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.026040 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-certs\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.026224 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.026208 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3815179-7ff3-4db0-b671-ecdafd4e977c-istiod-ca-cert\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.027144 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.027116 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3815179-7ff3-4db0-b671-ecdafd4e977c-istiod-ca-cert\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.031471 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.028002 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-data\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.031633 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.028334 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-certs\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.041661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.028867 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-socket\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.041661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.029105 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-credential-socket\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.041885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.033533 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-podinfo\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.041885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.041545 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-envoy\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.086931 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.086855 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcnbq\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-kube-api-access-rcnbq\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.087224 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.087204 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-token\") pod \"e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.140099 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.140065 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:33.567906 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.567877 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z"] Oct 31 01:12:33.568049 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:12:33.567999 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3815179_7ff3_4db0_b671_ecdafd4e977c.slice/crio-120e1e49298dcc6a3f43a242856a2ef5726e9f342281be4136e53fc92b497c33 WatchSource:0}: Error finding container 120e1e49298dcc6a3f43a242856a2ef5726e9f342281be4136e53fc92b497c33: Status 404 returned error can't find the container with id 120e1e49298dcc6a3f43a242856a2ef5726e9f342281be4136e53fc92b497c33 Oct 31 01:12:33.570557 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.570524 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:12:33.570648 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.570588 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:12:33.570648 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.570619 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:12:33.648932 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:33.648898 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" event={"ID":"d3815179-7ff3-4db0-b671-ecdafd4e977c","Type":"ContainerStarted","Data":"120e1e49298dcc6a3f43a242856a2ef5726e9f342281be4136e53fc92b497c33"} Oct 31 01:12:34.658984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:34.658926 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" event={"ID":"d3815179-7ff3-4db0-b671-ecdafd4e977c","Type":"ContainerStarted","Data":"ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b"} Oct 31 01:12:34.825163 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:34.825111 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podStartSLOduration=2.825096585 podStartE2EDuration="2.825096585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:34.823355844 +0000 UTC m=+2647.025606276" watchObservedRunningTime="2026-10-31 01:12:34.825096585 +0000 UTC m=+2647.027347095" Oct 31 01:12:35.141100 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:35.141057 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:35.147003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:35.146977 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:35.667405 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:35.667373 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:35.668489 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:12:35.668464 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:12:37.018502 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:12:37.018459 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:13:07.019853 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:13:07.019811 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:13:15.055137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.055093 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2"] Oct 31 01:13:15.059984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.059952 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.113569 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.113536 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2"] Oct 31 01:13:15.156850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.156816 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-66e263\"" Oct 31 01:13:15.156850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.156825 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-66e263-dockercfg-sj8jg\"" Oct 31 01:13:15.199025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.198990 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.199240 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.199218 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnq8z\" (UniqueName: \"kubernetes.io/projected/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-kube-api-access-hnq8z\") pod \"payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.300395 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.300358 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.300617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.300599 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnq8z\" (UniqueName: \"kubernetes.io/projected/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-kube-api-access-hnq8z\") pod \"payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.301173 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.301153 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.360327 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.360253 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnq8z\" (UniqueName: \"kubernetes.io/projected/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-kube-api-access-hnq8z\") pod \"payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.378827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.378773 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:15.754101 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.754057 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2"] Oct 31 01:13:15.754596 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:13:15.754575 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffee017_ab05_458b_bd42_92ef9eb9fd0d.slice/crio-c404092a16b8070d3bfd87b2f11276fb8418bf3603e3d2f2c02db5e8fb09740c WatchSource:0}: Error finding container c404092a16b8070d3bfd87b2f11276fb8418bf3603e3d2f2c02db5e8fb09740c: Status 404 returned error can't find the container with id c404092a16b8070d3bfd87b2f11276fb8418bf3603e3d2f2c02db5e8fb09740c Oct 31 01:13:15.866820 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.866776 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2"] Oct 31 01:13:15.871048 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.871025 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:15.925042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:15.925004 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2"] Oct 31 01:13:16.003909 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.003816 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" event={"ID":"6ffee017-ab05-458b-bd42-92ef9eb9fd0d","Type":"ContainerStarted","Data":"be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64"} Oct 31 01:13:16.003909 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.003878 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" event={"ID":"6ffee017-ab05-458b-bd42-92ef9eb9fd0d","Type":"ContainerStarted","Data":"c404092a16b8070d3bfd87b2f11276fb8418bf3603e3d2f2c02db5e8fb09740c"} Oct 31 01:13:16.003909 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.003904 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:16.009691 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.009667 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshvt\" (UniqueName: \"kubernetes.io/projected/b067542f-0858-478b-b4b8-249b27d805a8-kube-api-access-qshvt\") pod \"payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:16.009904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.009878 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b067542f-0858-478b-b4b8-249b27d805a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:16.111026 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.110991 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshvt\" (UniqueName: \"kubernetes.io/projected/b067542f-0858-478b-b4b8-249b27d805a8-kube-api-access-qshvt\") pod \"payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:16.111502 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.111125 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b067542f-0858-478b-b4b8-249b27d805a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:16.111848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.111823 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b067542f-0858-478b-b4b8-249b27d805a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:16.139617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.139559 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" podStartSLOduration=1.139543955 podStartE2EDuration="1.139543955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:16.137679288 +0000 UTC m=+2688.339929721" watchObservedRunningTime="2026-10-31 01:13:16.139543955 +0000 UTC m=+2688.341794386" Oct 31 01:13:16.184251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.184218 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshvt\" (UniqueName: \"kubernetes.io/projected/b067542f-0858-478b-b4b8-249b27d805a8-kube-api-access-qshvt\") pod \"payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:16.185250 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.185222 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:16.560384 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:16.560346 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2"] Oct 31 01:13:16.561244 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:13:16.561218 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb067542f_0858_478b_b4b8_249b27d805a8.slice/crio-8beab89ccb5690db36ab635e4c5c41228b37a5ede64a1eaf1ec3e9f0b7bd5dda WatchSource:0}: Error finding container 8beab89ccb5690db36ab635e4c5c41228b37a5ede64a1eaf1ec3e9f0b7bd5dda: Status 404 returned error can't find the container with id 8beab89ccb5690db36ab635e4c5c41228b37a5ede64a1eaf1ec3e9f0b7bd5dda Oct 31 01:13:17.015860 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:17.015815 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" event={"ID":"b067542f-0858-478b-b4b8-249b27d805a8","Type":"ContainerStarted","Data":"2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029"} Oct 31 01:13:17.016055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:17.015891 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" event={"ID":"b067542f-0858-478b-b4b8-249b27d805a8","Type":"ContainerStarted","Data":"8beab89ccb5690db36ab635e4c5c41228b37a5ede64a1eaf1ec3e9f0b7bd5dda"} Oct 31 01:13:17.016249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:17.016227 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:17.134916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:17.134853 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" podStartSLOduration=2.1348339259999998 podStartE2EDuration="2.134833926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:17.134587743 +0000 UTC m=+2689.336838178" watchObservedRunningTime="2026-10-31 01:13:17.134833926 +0000 UTC m=+2689.337084361" Oct 31 01:13:27.017862 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:27.017827 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:13:28.026411 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:28.026356 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:13:28.884039 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:28.884010 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:13:29.000350 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:29.000317 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:13:37.019442 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:13:37.019401 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:13:37.132682 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:13:37.132652 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2_b067542f-0858-478b-b4b8-249b27d805a8/payload-processing/0.log" Oct 31 01:14:07.019744 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:14:07.019701 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:14:19.067984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:19.067934 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:14:19.068498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:19.068002 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:20.533650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:20.533567 5890 generic.go:356] "Generic (PLEG): container finished" podID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerID="ff65640f1bede847685b9430dc25365b9efea422c10fc9a54b5a99e3536fb5dc" exitCode=0 Oct 31 01:14:20.533650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:20.533638 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" event={"ID":"1a9096a8-7cf2-4454-a1c3-73cea55b63fc","Type":"ContainerDied","Data":"ff65640f1bede847685b9430dc25365b9efea422c10fc9a54b5a99e3536fb5dc"} Oct 31 01:14:20.534132 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:20.534113 5890 scope.go:122] "RemoveContainer" containerID="ff65640f1bede847685b9430dc25365b9efea422c10fc9a54b5a99e3536fb5dc" Oct 31 01:14:20.535818 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:20.535776 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:20.535914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:20.535868 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:20.535914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:20.535905 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:21.543065 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:21.543031 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" event={"ID":"1a9096a8-7cf2-4454-a1c3-73cea55b63fc","Type":"ContainerStarted","Data":"3781eb0cf69af4a3ccee5f574463d1075a3aac99a12b1fc2e1d00d4494e5c5b2"} Oct 31 01:14:21.922362 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:21.922329 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:14:21.922643 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:21.922621 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 01:14:21.922696 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:21.922682 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 01:14:22.922484 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:22.922445 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 01:14:22.922870 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:22.922523 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 01:14:23.922415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:23.922376 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 01:14:23.922589 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:23.922440 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 01:14:24.924264 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:24.924209 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 01:14:24.925009 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:24.924976 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 01:14:25.922576 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:25.922544 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 31 01:14:25.922849 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:25.922826 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 31 01:14:27.011990 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:27.011954 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:14:27.012736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:27.012706 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:14:27.013527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:27.013501 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:14:35.669472 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.669430 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 01:14:35.670040 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.669488 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 01:14:35.685321 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.685286 5890 generic.go:356] "Generic (PLEG): container finished" podID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerID="ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b" exitCode=0 Oct 31 01:14:35.685477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.685357 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" event={"ID":"d3815179-7ff3-4db0-b671-ecdafd4e977c","Type":"ContainerDied","Data":"ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b"} Oct 31 01:14:35.685882 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.685864 5890 scope.go:122] "RemoveContainer" containerID="ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b" Oct 31 01:14:35.687506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.687474 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:35.687602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.687546 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:35.687602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:35.687577 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:36.705675 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:36.704871 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" event={"ID":"d3815179-7ff3-4db0-b671-ecdafd4e977c","Type":"ContainerStarted","Data":"da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0"} Oct 31 01:14:37.021141 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:14:37.020965 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:14:37.140466 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:37.140436 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:14:37.140642 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:37.140544 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 01:14:37.140642 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:37.140588 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 01:14:38.140728 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:38.140689 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 01:14:38.141256 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:38.140754 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 01:14:39.141128 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:39.141091 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 01:14:39.141558 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:39.141148 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 01:14:40.141172 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:40.141136 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 01:14:40.141580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:40.141223 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 01:14:41.140542 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:41.140508 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 01:14:41.140749 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:41.140564 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 01:14:42.140687 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:42.140652 5890 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 01:14:42.141190 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:42.140705 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 01:14:43.147397 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:43.147360 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:14:43.148025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:43.148003 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:14:43.150306 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:43.150280 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:14:45.139861 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.139824 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr"] Oct 31 01:14:45.140361 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.140027 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" containerID="cri-o://46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150" gracePeriod=30 Oct 31 01:14:45.188283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.188253 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql"] Oct 31 01:14:45.188474 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.188453 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" containerID="cri-o://47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2" gracePeriod=30 Oct 31 01:14:45.192904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.192877 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs"] Oct 31 01:14:45.193120 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.193099 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" containerID="cri-o://f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5" gracePeriod=30 Oct 31 01:14:45.235311 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.235281 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9"] Oct 31 01:14:45.235485 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:45.235462 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" containerID="cri-o://ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb" gracePeriod=30 Oct 31 01:14:50.482497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.482034 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:14:50.520938 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.520903 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-envoy\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521067 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-certs\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521097 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-podinfo\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521236 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istiod-ca-cert\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521312 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521276 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-credential-socket\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521365 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521323 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-token\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-token\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521362 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-data\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521421 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-kube-api-access-pzzqq\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-kube-api-access-pzzqq\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.521487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.521462 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-socket\") pod \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\" (UID: \"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e\") " Oct 31 01:14:50.522601 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.522570 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-data" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.523248 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.522913 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-credential-socket" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.523248 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.523214 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istiod-ca-cert" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:50.523484 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.523457 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-certs" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.523599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.523577 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-socket" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.524918 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.524798 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-podinfo" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:50.537192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.537150 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-envoy" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.537192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.537167 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-kube-api-access-pzzqq" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "kube-api-access-pzzqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.539400 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.539369 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-token" pod "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" (UID: "9d5d6be7-ff57-4a2c-a7d4-2b298679c55e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.561976 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.561945 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:14:50.573032 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.573005 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:14:50.597894 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.597871 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:14:50.622814 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.622755 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-token\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-token\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.622931 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.622841 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-certs\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.622931 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.622874 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-kube-api-access-jz2kq\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-kube-api-access-jz2kq\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.622931 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.622898 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-envoy\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.622931 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.622919 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-socket\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.623105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.622983 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-data\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623021 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-envoy\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623053 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-istio-token\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-istio-token\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.623105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623085 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-data\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.623410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623185 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-credential-socket\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.623410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623228 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-data\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.623410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623259 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-socket" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.623410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623271 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-certs\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.623410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623289 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-data" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.623410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623337 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f5f024-0884-40ea-af34-956365138271-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f5f024-0884-40ea-af34-956365138271-istiod-ca-cert\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.623410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623396 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-socket\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623451 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-certs" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623456 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63f5f024-0884-40ea-af34-956365138271-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f5f024-0884-40ea-af34-956365138271-istio-podinfo\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623508 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-credential-socket\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623515 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-certs" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623536 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a0f9e8-65df-441a-821e-952dca1a4641-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a0f9e8-65df-441a-821e-952dca1a4641-istiod-ca-cert\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623586 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-kube-api-access-xmt6z\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-kube-api-access-xmt6z\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623617 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-credential-socket\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623688 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-socket" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623693 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-istio-token\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-istio-token\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623728 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-podinfo\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623765 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-certs\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.623829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623816 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-kube-api-access-6qfwt\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-kube-api-access-6qfwt\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.624475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623867 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-socket\") pod \"63f5f024-0884-40ea-af34-956365138271\" (UID: \"63f5f024-0884-40ea-af34-956365138271\") " Oct 31 01:14:50.624475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623900 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-envoy\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.624475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623932 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1e5e5e-249e-453b-92d1-cb12ab095404-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1e5e5e-249e-453b-92d1-cb12ab095404-istiod-ca-cert\") pod \"2b1e5e5e-249e-453b-92d1-cb12ab095404\" (UID: \"2b1e5e5e-249e-453b-92d1-cb12ab095404\") " Oct 31 01:14:50.624475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.623966 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08a0f9e8-65df-441a-821e-952dca1a4641-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a0f9e8-65df-441a-821e-952dca1a4641-istio-podinfo\") pod \"08a0f9e8-65df-441a-821e-952dca1a4641\" (UID: \"08a0f9e8-65df-441a-821e-952dca1a4641\") " Oct 31 01:14:50.624475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624004 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f5f024-0884-40ea-af34-956365138271-istiod-ca-cert" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:50.624475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624238 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-credential-socket" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624676 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624701 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624716 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzzqq\" (UniqueName: \"kubernetes.io/projected/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-kube-api-access-pzzqq\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624731 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624745 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624758 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624771 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624800 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624814 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624830 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624847 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624863 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624879 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f5f024-0884-40ea-af34-956365138271-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624894 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624908 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.624936 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.624923 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.625975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.625158 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-data" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.625975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.625402 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-data" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.625975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.625705 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-istio-token" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.625975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.625710 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-credential-socket" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.625975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.625927 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-socket" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.626248 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626125 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-certs" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.626248 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626189 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-envoy" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.626371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626274 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-token" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.626371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626320 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-kube-api-access-jz2kq" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "kube-api-access-jz2kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.627095 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626511 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08a0f9e8-65df-441a-821e-952dca1a4641-istio-podinfo" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:50.627095 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626548 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-credential-socket" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.627095 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626614 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63f5f024-0884-40ea-af34-956365138271-istio-podinfo" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:50.627095 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.626950 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a0f9e8-65df-441a-821e-952dca1a4641-istiod-ca-cert" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:50.627095 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.627035 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1e5e5e-249e-453b-92d1-cb12ab095404-istiod-ca-cert" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:50.628102 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.628076 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-podinfo" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:50.628199 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.628176 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-kube-api-access-xmt6z" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "kube-api-access-xmt6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.628318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.628301 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-istio-token" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.628611 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.628585 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-envoy" pod "08a0f9e8-65df-441a-821e-952dca1a4641" (UID: "08a0f9e8-65df-441a-821e-952dca1a4641"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.628673 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.628620 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-kube-api-access-6qfwt" pod "63f5f024-0884-40ea-af34-956365138271" (UID: "63f5f024-0884-40ea-af34-956365138271"). InnerVolumeSpecName "kube-api-access-6qfwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:50.628713 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.628689 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-envoy" pod "2b1e5e5e-249e-453b-92d1-cb12ab095404" (UID: "2b1e5e5e-249e-453b-92d1-cb12ab095404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:50.726575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726501 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726529 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726540 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726549 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726558 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f5f024-0884-40ea-af34-956365138271-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726566 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726575 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a0f9e8-65df-441a-821e-952dca1a4641-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726583 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmt6z\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-kube-api-access-xmt6z\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726593 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726601 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a0f9e8-65df-441a-821e-952dca1a4641-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726609 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726617 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a0f9e8-65df-441a-821e-952dca1a4641-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726626 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qfwt\" (UniqueName: \"kubernetes.io/projected/63f5f024-0884-40ea-af34-956365138271-kube-api-access-6qfwt\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726633 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726641 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726649 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1e5e5e-249e-453b-92d1-cb12ab095404-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726657 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a0f9e8-65df-441a-821e-952dca1a4641-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726665 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726673 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz2kq\" (UniqueName: \"kubernetes.io/projected/2b1e5e5e-249e-453b-92d1-cb12ab095404-kube-api-access-jz2kq\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.726942 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.726682 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f5f024-0884-40ea-af34-956365138271-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:50.805404 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.805357 5890 generic.go:356] "Generic (PLEG): container finished" podID="08a0f9e8-65df-441a-821e-952dca1a4641" containerID="47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2" exitCode=0 Oct 31 01:14:50.805595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.805420 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" Oct 31 01:14:50.805595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.805444 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" event={"ID":"08a0f9e8-65df-441a-821e-952dca1a4641","Type":"ContainerDied","Data":"47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2"} Oct 31 01:14:50.805595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.805506 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql" event={"ID":"08a0f9e8-65df-441a-821e-952dca1a4641","Type":"ContainerDied","Data":"dcb378dcb94b04ff66664ccc3b5c952ba8a48ae660979d057a9cd357fd02d1f5"} Oct 31 01:14:50.805595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.805541 5890 scope.go:122] "RemoveContainer" containerID="47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2" Oct 31 01:14:50.807487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.807278 5890 generic.go:356] "Generic (PLEG): container finished" podID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerID="46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150" exitCode=0 Oct 31 01:14:50.807487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.807395 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" Oct 31 01:14:50.807487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.807415 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" event={"ID":"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e","Type":"ContainerDied","Data":"46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150"} Oct 31 01:14:50.807487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.807455 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr" event={"ID":"9d5d6be7-ff57-4a2c-a7d4-2b298679c55e","Type":"ContainerDied","Data":"94e99a094674d7b88be06939692ff9e818d9093295a8a5d4fbfcaae08f64b298"} Oct 31 01:14:50.809335 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.809292 5890 generic.go:356] "Generic (PLEG): container finished" podID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerID="ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb" exitCode=0 Oct 31 01:14:50.809483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.809451 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" event={"ID":"2b1e5e5e-249e-453b-92d1-cb12ab095404","Type":"ContainerDied","Data":"ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb"} Oct 31 01:14:50.809565 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.809489 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" event={"ID":"2b1e5e5e-249e-453b-92d1-cb12ab095404","Type":"ContainerDied","Data":"2f8f0a4beea06f778f2b62c6981ce34b73751e57bdfd72f7170d067d23dc3b7f"} Oct 31 01:14:50.809565 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.809554 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9" Oct 31 01:14:50.812497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.812474 5890 generic.go:356] "Generic (PLEG): container finished" podID="63f5f024-0884-40ea-af34-956365138271" containerID="f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5" exitCode=0 Oct 31 01:14:50.812620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.812500 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" event={"ID":"63f5f024-0884-40ea-af34-956365138271","Type":"ContainerDied","Data":"f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5"} Oct 31 01:14:50.812620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.812525 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" Oct 31 01:14:50.812620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.812557 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs" event={"ID":"63f5f024-0884-40ea-af34-956365138271","Type":"ContainerDied","Data":"894acb9f73f5935f0b079fc92565cdb400c8de0d4077856f4ea9a2b6e78d1a3b"} Oct 31 01:14:50.820368 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.820349 5890 scope.go:122] "RemoveContainer" containerID="47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2" Oct 31 01:14:50.820656 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:14:50.820628 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2\": container with ID starting with 47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2 not found: ID does not exist" containerID="47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2" Oct 31 01:14:50.820810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.820677 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2"} err="failed to get container status \"47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2\": rpc error: code = NotFound desc = could not find container \"47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2\": container with ID starting with 47196dfeb12c56b70682e7fbffa429cb9a652d7a1fda6e8252fc32dbb41cf3c2 not found: ID does not exist" Oct 31 01:14:50.820810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.820702 5890 scope.go:122] "RemoveContainer" containerID="46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150" Oct 31 01:14:50.831811 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.831778 5890 scope.go:122] "RemoveContainer" containerID="46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150" Oct 31 01:14:50.832070 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:14:50.832052 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150\": container with ID starting with 46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150 not found: ID does not exist" containerID="46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150" Oct 31 01:14:50.832122 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.832077 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150"} err="failed to get container status \"46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150\": rpc error: code = NotFound desc = could not find container \"46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150\": container with ID starting with 46a670a4a2a9cfdf97b522fa2402b6db23a187478b81e42077526986a1385150 not found: ID does not exist" Oct 31 01:14:50.832122 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.832092 5890 scope.go:122] "RemoveContainer" containerID="ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb" Oct 31 01:14:50.841299 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.841284 5890 scope.go:122] "RemoveContainer" containerID="ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb" Oct 31 01:14:50.841558 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:14:50.841542 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb\": container with ID starting with ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb not found: ID does not exist" containerID="ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb" Oct 31 01:14:50.841607 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.841565 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb"} err="failed to get container status \"ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb\": rpc error: code = NotFound desc = could not find container \"ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb\": container with ID starting with ad1d27c848d8e312c383a7b612561610195251aef175bcf38088ea63e5cfa8eb not found: ID does not exist" Oct 31 01:14:50.841607 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.841583 5890 scope.go:122] "RemoveContainer" containerID="f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5" Oct 31 01:14:50.850283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.850266 5890 scope.go:122] "RemoveContainer" containerID="f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5" Oct 31 01:14:50.850551 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:14:50.850532 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5\": container with ID starting with f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5 not found: ID does not exist" containerID="f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5" Oct 31 01:14:50.850594 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:50.850559 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5"} err="failed to get container status \"f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5\": rpc error: code = NotFound desc = could not find container \"f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5\": container with ID starting with f54f8c39a0c345a1b21eec0a873e1d8f075e68addde00949df54d0a3ace97cf5 not found: ID does not exist" Oct 31 01:14:51.111987 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.111887 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs"] Oct 31 01:14:51.115351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.115320 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-kj2xs"] Oct 31 01:14:51.220431 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.220397 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql"] Oct 31 01:14:51.225368 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.225335 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-gqrql"] Oct 31 01:14:51.336526 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.336447 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr"] Oct 31 01:14:51.347404 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.347372 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w2wwr"] Oct 31 01:14:51.451444 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.451405 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9"] Oct 31 01:14:51.456403 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.456365 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-j8sv9"] Oct 31 01:14:51.933238 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.933202 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" path="/var/lib/kubelet/pods/08a0f9e8-65df-441a-821e-952dca1a4641/volumes" Oct 31 01:14:51.933739 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.933697 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" path="/var/lib/kubelet/pods/2b1e5e5e-249e-453b-92d1-cb12ab095404/volumes" Oct 31 01:14:51.934166 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.934149 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f5f024-0884-40ea-af34-956365138271" path="/var/lib/kubelet/pods/63f5f024-0884-40ea-af34-956365138271/volumes" Oct 31 01:14:51.935075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:14:51.935054 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" path="/var/lib/kubelet/pods/9d5d6be7-ff57-4a2c-a7d4-2b298679c55e/volumes" Oct 31 01:15:00.249515 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.249479 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6"] Oct 31 01:15:00.250356 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250334 5890 cpu_manager.go:414] "containerMap: removing container" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" Oct 31 01:15:00.250415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250359 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" Oct 31 01:15:00.250415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250395 5890 cpu_manager.go:414] "containerMap: removing container" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" Oct 31 01:15:00.250415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250403 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" Oct 31 01:15:00.250509 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250432 5890 cpu_manager.go:414] "containerMap: removing container" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" Oct 31 01:15:00.250509 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250439 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" Oct 31 01:15:00.250509 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250459 5890 cpu_manager.go:414] "containerMap: removing container" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" Oct 31 01:15:00.250509 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250467 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" Oct 31 01:15:00.250627 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250595 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5d6be7-ff57-4a2c-a7d4-2b298679c55e" containerName="istio-proxy" Oct 31 01:15:00.250627 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250609 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1e5e5e-249e-453b-92d1-cb12ab095404" containerName="istio-proxy" Oct 31 01:15:00.250627 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250623 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f5f024-0884-40ea-af34-956365138271" containerName="istio-proxy" Oct 31 01:15:00.250725 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.250635 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a0f9e8-65df-441a-821e-952dca1a4641" containerName="istio-proxy" Oct 31 01:15:00.253546 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.253523 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v"] Oct 31 01:15:00.253681 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.253660 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.257111 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.257088 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.275369 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.275341 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v"] Oct 31 01:15:00.279020 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.278999 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.314639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.314599 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6"] Oct 31 01:15:00.324379 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.324348 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v"] Oct 31 01:15:00.349819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.349747 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.349988 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.349959 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-85bp6\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.350070 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.350014 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f51c1e8-1899-4832-960d-ab98cbd588f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.350210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.350182 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldpm\" (UniqueName: \"kubernetes.io/projected/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-kube-api-access-9ldpm\") pod \"maas-api-key-cleanup-29890155-85bp6\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.350268 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.350224 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-kube-api-access-z9j2q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.350329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.350293 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxw4q\" (UniqueName: \"kubernetes.io/projected/6f51c1e8-1899-4832-960d-ab98cbd588f8-kube-api-access-hxw4q\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.371896 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.371862 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:15:00.382310 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.382275 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v"] Oct 31 01:15:00.452148 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.452107 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldpm\" (UniqueName: \"kubernetes.io/projected/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-kube-api-access-9ldpm\") pod \"maas-api-key-cleanup-29890155-85bp6\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.452345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.452155 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-kube-api-access-z9j2q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.452345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.452193 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxw4q\" (UniqueName: \"kubernetes.io/projected/6f51c1e8-1899-4832-960d-ab98cbd588f8-kube-api-access-hxw4q\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.452345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.452272 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.452536 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.452409 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-85bp6\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.452536 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.452473 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f51c1e8-1899-4832-960d-ab98cbd588f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.453246 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.453216 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.453398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.453216 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-85bp6\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.453398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.453271 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f51c1e8-1899-4832-960d-ab98cbd588f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.512735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.512658 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldpm\" (UniqueName: \"kubernetes.io/projected/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-kube-api-access-9ldpm\") pod \"maas-api-key-cleanup-29890155-85bp6\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.513498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.513473 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-kube-api-access-z9j2q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.513836 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.513778 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxw4q\" (UniqueName: \"kubernetes.io/projected/6f51c1e8-1899-4832-960d-ab98cbd588f8-kube-api-access-hxw4q\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.569523 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.569491 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:00.578260 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.578232 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:00.593651 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.593594 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:00.954639 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:15:00.954604 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49433dfd_fe62_48a1_90ff_e4fabd2d2c79.slice/crio-fd61ccd26af0df7f282abf72bfdf7bcfe6fe43a091808a6e8d03122095074ec6 WatchSource:0}: Error finding container fd61ccd26af0df7f282abf72bfdf7bcfe6fe43a091808a6e8d03122095074ec6: Status 404 returned error can't find the container with id fd61ccd26af0df7f282abf72bfdf7bcfe6fe43a091808a6e8d03122095074ec6 Oct 31 01:15:00.954853 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:00.954829 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6"] Oct 31 01:15:01.000463 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.000434 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v"] Oct 31 01:15:01.001025 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:15:01.000998 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4a8cc1_c449_42fa_8b0b_4c614b8935ae.slice/crio-46729772d4f5e4608b780e803a27e4de43ebabaedf1f2fb832aa570946ed3427 WatchSource:0}: Error finding container 46729772d4f5e4608b780e803a27e4de43ebabaedf1f2fb832aa570946ed3427: Status 404 returned error can't find the container with id 46729772d4f5e4608b780e803a27e4de43ebabaedf1f2fb832aa570946ed3427 Oct 31 01:15:01.002826 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:15:01.002803 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f51c1e8_1899_4832_960d_ab98cbd588f8.slice/crio-640e5d170010673ad53a8b18bd0cfb7c1a784e0385af44ea9b08c03acfd863ff WatchSource:0}: Error finding container 640e5d170010673ad53a8b18bd0cfb7c1a784e0385af44ea9b08c03acfd863ff: Status 404 returned error can't find the container with id 640e5d170010673ad53a8b18bd0cfb7c1a784e0385af44ea9b08c03acfd863ff Oct 31 01:15:01.003587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.003563 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v"] Oct 31 01:15:01.905868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.905810 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" event={"ID":"6f51c1e8-1899-4832-960d-ab98cbd588f8","Type":"ContainerStarted","Data":"8c8a5180acb7c944869dd5d54b1d7077a514d34f30ab02ad2d0f02c887b1af41"} Oct 31 01:15:01.906345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.905885 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" event={"ID":"6f51c1e8-1899-4832-960d-ab98cbd588f8","Type":"ContainerStarted","Data":"640e5d170010673ad53a8b18bd0cfb7c1a784e0385af44ea9b08c03acfd863ff"} Oct 31 01:15:01.907502 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.907469 5890 generic.go:356] "Generic (PLEG): container finished" podID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerID="de1daf45e51c0c5c2b5061a28b1beac0ba562dbd2ecab4cb401ddcb3beb8984f" exitCode=60 Oct 31 01:15:01.907643 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.907559 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" event={"ID":"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae","Type":"ContainerDied","Data":"de1daf45e51c0c5c2b5061a28b1beac0ba562dbd2ecab4cb401ddcb3beb8984f"} Oct 31 01:15:01.907643 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.907611 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" event={"ID":"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae","Type":"ContainerStarted","Data":"46729772d4f5e4608b780e803a27e4de43ebabaedf1f2fb832aa570946ed3427"} Oct 31 01:15:01.907831 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.907818 5890 scope.go:122] "RemoveContainer" containerID="de1daf45e51c0c5c2b5061a28b1beac0ba562dbd2ecab4cb401ddcb3beb8984f" Oct 31 01:15:01.909683 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.909658 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerStarted","Data":"6ab213b8e56ba9c7f5d9b1f37e0a7716f8b6c0497d5b538f3fd1b684892ffee8"} Oct 31 01:15:01.909871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:01.909715 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerStarted","Data":"fd61ccd26af0df7f282abf72bfdf7bcfe6fe43a091808a6e8d03122095074ec6"} Oct 31 01:15:02.028541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.028480 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" podStartSLOduration=2.028466272 podStartE2EDuration="2.028466272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.026464239 +0000 UTC m=+2794.228714671" watchObservedRunningTime="2026-10-31 01:15:02.028466272 +0000 UTC m=+2794.230716710" Oct 31 01:15:02.918735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.918697 5890 generic.go:356] "Generic (PLEG): container finished" podID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerID="8c8a5180acb7c944869dd5d54b1d7077a514d34f30ab02ad2d0f02c887b1af41" exitCode=60 Oct 31 01:15:02.919202 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.918813 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" event={"ID":"6f51c1e8-1899-4832-960d-ab98cbd588f8","Type":"ContainerDied","Data":"8c8a5180acb7c944869dd5d54b1d7077a514d34f30ab02ad2d0f02c887b1af41"} Oct 31 01:15:02.919202 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.919111 5890 scope.go:122] "RemoveContainer" containerID="8c8a5180acb7c944869dd5d54b1d7077a514d34f30ab02ad2d0f02c887b1af41" Oct 31 01:15:02.922555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.922532 5890 generic.go:356] "Generic (PLEG): container finished" podID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerID="de21843ec5710a6773ba9bf774830b04fa9ec3974c2c5ed301627838f267b72b" exitCode=60 Oct 31 01:15:02.922632 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.922617 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" event={"ID":"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae","Type":"ContainerDied","Data":"de21843ec5710a6773ba9bf774830b04fa9ec3974c2c5ed301627838f267b72b"} Oct 31 01:15:02.922892 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.922673 5890 scope.go:122] "RemoveContainer" containerID="de1daf45e51c0c5c2b5061a28b1beac0ba562dbd2ecab4cb401ddcb3beb8984f" Oct 31 01:15:02.923035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.923019 5890 scope.go:122] "RemoveContainer" containerID="de21843ec5710a6773ba9bf774830b04fa9ec3974c2c5ed301627838f267b72b" Oct 31 01:15:02.923279 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:02.923258 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v_odh-ai-gateway-infra(9b4a8cc1-c449-42fa-8b0b-4c614b8935ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" Oct 31 01:15:02.924760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.924740 5890 generic.go:356] "Generic (PLEG): container finished" podID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerID="6ab213b8e56ba9c7f5d9b1f37e0a7716f8b6c0497d5b538f3fd1b684892ffee8" exitCode=60 Oct 31 01:15:02.924899 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.924844 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerDied","Data":"6ab213b8e56ba9c7f5d9b1f37e0a7716f8b6c0497d5b538f3fd1b684892ffee8"} Oct 31 01:15:02.925115 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:02.925099 5890 scope.go:122] "RemoveContainer" containerID="6ab213b8e56ba9c7f5d9b1f37e0a7716f8b6c0497d5b538f3fd1b684892ffee8" Oct 31 01:15:03.047961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:03.047911 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" podStartSLOduration=3.047899037 podStartE2EDuration="3.047899037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.283082581 +0000 UTC m=+2794.485333017" watchObservedRunningTime="2026-10-31 01:15:03.047899037 +0000 UTC m=+2795.250149539" Oct 31 01:15:03.937151 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:03.937106 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" event={"ID":"6f51c1e8-1899-4832-960d-ab98cbd588f8","Type":"ContainerStarted","Data":"21f7dbc2f1e8cd6ca655ca8ef6b215c053fb5303a3344325996adbfa3826725c"} Oct 31 01:15:03.939039 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:03.939003 5890 scope.go:122] "RemoveContainer" containerID="de21843ec5710a6773ba9bf774830b04fa9ec3974c2c5ed301627838f267b72b" Oct 31 01:15:03.939340 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:03.939310 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v_odh-ai-gateway-infra(9b4a8cc1-c449-42fa-8b0b-4c614b8935ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" Oct 31 01:15:03.940843 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:03.940814 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerStarted","Data":"883bdac8fa3eee999bc86c027854f743a03d7fab0d67c2617b6228cd4537acc2"} Oct 31 01:15:04.951185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.951147 5890 generic.go:356] "Generic (PLEG): container finished" podID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerID="21f7dbc2f1e8cd6ca655ca8ef6b215c053fb5303a3344325996adbfa3826725c" exitCode=60 Oct 31 01:15:04.951641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.951192 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" event={"ID":"6f51c1e8-1899-4832-960d-ab98cbd588f8","Type":"ContainerDied","Data":"21f7dbc2f1e8cd6ca655ca8ef6b215c053fb5303a3344325996adbfa3826725c"} Oct 31 01:15:04.951641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.951260 5890 scope.go:122] "RemoveContainer" containerID="8c8a5180acb7c944869dd5d54b1d7077a514d34f30ab02ad2d0f02c887b1af41" Oct 31 01:15:04.951641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.951546 5890 scope.go:122] "RemoveContainer" containerID="21f7dbc2f1e8cd6ca655ca8ef6b215c053fb5303a3344325996adbfa3826725c" Oct 31 01:15:04.952032 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:04.951886 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v_odh-ai-gateway-infra(6f51c1e8-1899-4832-960d-ab98cbd588f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" Oct 31 01:15:04.953605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.953578 5890 generic.go:356] "Generic (PLEG): container finished" podID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerID="883bdac8fa3eee999bc86c027854f743a03d7fab0d67c2617b6228cd4537acc2" exitCode=60 Oct 31 01:15:04.953747 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.953670 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerDied","Data":"883bdac8fa3eee999bc86c027854f743a03d7fab0d67c2617b6228cd4537acc2"} Oct 31 01:15:04.953979 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.953960 5890 scope.go:122] "RemoveContainer" containerID="883bdac8fa3eee999bc86c027854f743a03d7fab0d67c2617b6228cd4537acc2" Oct 31 01:15:04.954255 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:04.954236 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-85bp6_odh-ai-gateway-infra(49433dfd-fe62-48a1-90ff-e4fabd2d2c79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" Oct 31 01:15:04.964316 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:04.964298 5890 scope.go:122] "RemoveContainer" containerID="6ab213b8e56ba9c7f5d9b1f37e0a7716f8b6c0497d5b538f3fd1b684892ffee8" Oct 31 01:15:05.963278 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:05.963250 5890 scope.go:122] "RemoveContainer" containerID="21f7dbc2f1e8cd6ca655ca8ef6b215c053fb5303a3344325996adbfa3826725c" Oct 31 01:15:05.963709 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:05.963545 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v_odh-ai-gateway-infra(6f51c1e8-1899-4832-960d-ab98cbd588f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" Oct 31 01:15:05.964885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:05.964865 5890 scope.go:122] "RemoveContainer" containerID="883bdac8fa3eee999bc86c027854f743a03d7fab0d67c2617b6228cd4537acc2" Oct 31 01:15:05.965156 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:05.965135 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-85bp6_odh-ai-gateway-infra(49433dfd-fe62-48a1-90ff-e4fabd2d2c79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" Oct 31 01:15:07.020230 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:07.020182 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:15:16.926282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:16.926201 5890 scope.go:122] "RemoveContainer" containerID="de21843ec5710a6773ba9bf774830b04fa9ec3974c2c5ed301627838f267b72b" Oct 31 01:15:17.934556 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:17.934522 5890 scope.go:122] "RemoveContainer" containerID="21f7dbc2f1e8cd6ca655ca8ef6b215c053fb5303a3344325996adbfa3826725c" Oct 31 01:15:17.934961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:17.934640 5890 scope.go:122] "RemoveContainer" containerID="883bdac8fa3eee999bc86c027854f743a03d7fab0d67c2617b6228cd4537acc2" Oct 31 01:15:18.061155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:18.061127 5890 generic.go:356] "Generic (PLEG): container finished" podID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerID="3ddf2f8b3d3e2e70c6525bbb089708f32303853aa38670a6b229152c4209e419" exitCode=60 Oct 31 01:15:18.061295 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:18.061200 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" event={"ID":"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae","Type":"ContainerDied","Data":"3ddf2f8b3d3e2e70c6525bbb089708f32303853aa38670a6b229152c4209e419"} Oct 31 01:15:18.061295 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:18.061264 5890 scope.go:122] "RemoveContainer" containerID="de21843ec5710a6773ba9bf774830b04fa9ec3974c2c5ed301627838f267b72b" Oct 31 01:15:18.061657 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:18.061636 5890 scope.go:122] "RemoveContainer" containerID="3ddf2f8b3d3e2e70c6525bbb089708f32303853aa38670a6b229152c4209e419" Oct 31 01:15:18.062047 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:18.062012 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v_odh-ai-gateway-infra(9b4a8cc1-c449-42fa-8b0b-4c614b8935ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" Oct 31 01:15:19.070928 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.070888 5890 generic.go:356] "Generic (PLEG): container finished" podID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerID="8889a1dc180ba7975ba63cff267ecfbb857279fbd322373785758fd29fb3f32d" exitCode=60 Oct 31 01:15:19.071421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.070959 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" event={"ID":"6f51c1e8-1899-4832-960d-ab98cbd588f8","Type":"ContainerDied","Data":"8889a1dc180ba7975ba63cff267ecfbb857279fbd322373785758fd29fb3f32d"} Oct 31 01:15:19.071421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.071080 5890 scope.go:122] "RemoveContainer" containerID="21f7dbc2f1e8cd6ca655ca8ef6b215c053fb5303a3344325996adbfa3826725c" Oct 31 01:15:19.071544 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.071523 5890 scope.go:122] "RemoveContainer" containerID="8889a1dc180ba7975ba63cff267ecfbb857279fbd322373785758fd29fb3f32d" Oct 31 01:15:19.071951 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:19.071909 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v_odh-ai-gateway-infra(6f51c1e8-1899-4832-960d-ab98cbd588f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" Oct 31 01:15:19.075486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.075453 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerStarted","Data":"87507c3962370a5a2b03a1577172811e456a74018323d5dfd6620d49eb20d8ce"} Oct 31 01:15:19.186906 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.186859 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v"] Oct 31 01:15:19.371006 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.370979 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:19.406694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.406663 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-maas-api-service-ca\") pod \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " Oct 31 01:15:19.406892 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.406714 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-kube-api-access-z9j2q\") pod \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\" (UID: \"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae\") " Oct 31 01:15:19.407045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.407021 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-maas-api-service-ca" pod "9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" (UID: "9b4a8cc1-c449-42fa-8b0b-4c614b8935ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.407345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.407325 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.408921 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.408902 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-kube-api-access-z9j2q" pod "9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" (UID: "9b4a8cc1-c449-42fa-8b0b-4c614b8935ae"). InnerVolumeSpecName "kube-api-access-z9j2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.508287 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:19.508197 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae-kube-api-access-z9j2q\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.088006 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.087977 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" Oct 31 01:15:20.088501 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.088007 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v" event={"ID":"9b4a8cc1-c449-42fa-8b0b-4c614b8935ae","Type":"ContainerDied","Data":"46729772d4f5e4608b780e803a27e4de43ebabaedf1f2fb832aa570946ed3427"} Oct 31 01:15:20.088501 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.088085 5890 scope.go:122] "RemoveContainer" containerID="3ddf2f8b3d3e2e70c6525bbb089708f32303853aa38670a6b229152c4209e419" Oct 31 01:15:20.091725 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.091202 5890 generic.go:356] "Generic (PLEG): container finished" podID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerID="87507c3962370a5a2b03a1577172811e456a74018323d5dfd6620d49eb20d8ce" exitCode=60 Oct 31 01:15:20.091725 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.091319 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerDied","Data":"87507c3962370a5a2b03a1577172811e456a74018323d5dfd6620d49eb20d8ce"} Oct 31 01:15:20.091725 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.091660 5890 scope.go:122] "RemoveContainer" containerID="87507c3962370a5a2b03a1577172811e456a74018323d5dfd6620d49eb20d8ce" Oct 31 01:15:20.092064 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:20.092036 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-85bp6_odh-ai-gateway-infra(49433dfd-fe62-48a1-90ff-e4fabd2d2c79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" Oct 31 01:15:20.103131 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.103109 5890 scope.go:122] "RemoveContainer" containerID="883bdac8fa3eee999bc86c027854f743a03d7fab0d67c2617b6228cd4537acc2" Oct 31 01:15:20.198524 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.198444 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v"] Oct 31 01:15:20.250245 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.250211 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v"] Oct 31 01:15:20.257192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.257161 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890155-27g4v"] Oct 31 01:15:20.323834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.313525 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6"] Oct 31 01:15:20.406209 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.406173 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:20.544365 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.544269 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f51c1e8-1899-4832-960d-ab98cbd588f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f51c1e8-1899-4832-960d-ab98cbd588f8-maas-api-service-ca\") pod \"6f51c1e8-1899-4832-960d-ab98cbd588f8\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " Oct 31 01:15:20.544542 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.544451 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f51c1e8-1899-4832-960d-ab98cbd588f8-kube-api-access-hxw4q\" (UniqueName: \"kubernetes.io/projected/6f51c1e8-1899-4832-960d-ab98cbd588f8-kube-api-access-hxw4q\") pod \"6f51c1e8-1899-4832-960d-ab98cbd588f8\" (UID: \"6f51c1e8-1899-4832-960d-ab98cbd588f8\") " Oct 31 01:15:20.544665 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.544637 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f51c1e8-1899-4832-960d-ab98cbd588f8-maas-api-service-ca" pod "6f51c1e8-1899-4832-960d-ab98cbd588f8" (UID: "6f51c1e8-1899-4832-960d-ab98cbd588f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.545015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.544988 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f51c1e8-1899-4832-960d-ab98cbd588f8-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.546670 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.546647 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f51c1e8-1899-4832-960d-ab98cbd588f8-kube-api-access-hxw4q" pod "6f51c1e8-1899-4832-960d-ab98cbd588f8" (UID: "6f51c1e8-1899-4832-960d-ab98cbd588f8"). InnerVolumeSpecName "kube-api-access-hxw4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.646270 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:20.646235 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxw4q\" (UniqueName: \"kubernetes.io/projected/6f51c1e8-1899-4832-960d-ab98cbd588f8-kube-api-access-hxw4q\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:21.106170 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.106136 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" event={"ID":"6f51c1e8-1899-4832-960d-ab98cbd588f8","Type":"ContainerDied","Data":"640e5d170010673ad53a8b18bd0cfb7c1a784e0385af44ea9b08c03acfd863ff"} Oct 31 01:15:21.106539 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.106175 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v" Oct 31 01:15:21.106539 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.106207 5890 scope.go:122] "RemoveContainer" containerID="8889a1dc180ba7975ba63cff267ecfbb857279fbd322373785758fd29fb3f32d" Oct 31 01:15:21.273956 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.273921 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk"] Oct 31 01:15:21.274211 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.274163 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" containerID="cri-o://7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf" gracePeriod=30 Oct 31 01:15:21.317810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.311075 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:21.334910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.334854 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v"] Oct 31 01:15:21.341250 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.341222 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890155-jgp9v"] Oct 31 01:15:21.457932 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.457896 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-kube-api-access-9ldpm\" (UniqueName: \"kubernetes.io/projected/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-kube-api-access-9ldpm\") pod \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " Oct 31 01:15:21.458130 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.457980 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-maas-api-service-ca\") pod \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\" (UID: \"49433dfd-fe62-48a1-90ff-e4fabd2d2c79\") " Oct 31 01:15:21.458344 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.458313 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-maas-api-service-ca" pod "49433dfd-fe62-48a1-90ff-e4fabd2d2c79" (UID: "49433dfd-fe62-48a1-90ff-e4fabd2d2c79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:21.458653 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.458630 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:21.460555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.460529 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-kube-api-access-9ldpm" pod "49433dfd-fe62-48a1-90ff-e4fabd2d2c79" (UID: "49433dfd-fe62-48a1-90ff-e4fabd2d2c79"). InnerVolumeSpecName "kube-api-access-9ldpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:21.560463 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.560427 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ldpm\" (UniqueName: \"kubernetes.io/projected/49433dfd-fe62-48a1-90ff-e4fabd2d2c79-kube-api-access-9ldpm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:21.932921 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.932887 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" path="/var/lib/kubelet/pods/6f51c1e8-1899-4832-960d-ab98cbd588f8/volumes" Oct 31 01:15:21.933348 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:21.933335 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" path="/var/lib/kubelet/pods/9b4a8cc1-c449-42fa-8b0b-4c614b8935ae/volumes" Oct 31 01:15:22.115606 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:22.115574 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" Oct 31 01:15:22.116085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:22.115612 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6" event={"ID":"49433dfd-fe62-48a1-90ff-e4fabd2d2c79","Type":"ContainerDied","Data":"fd61ccd26af0df7f282abf72bfdf7bcfe6fe43a091808a6e8d03122095074ec6"} Oct 31 01:15:22.116085 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:22.115679 5890 scope.go:122] "RemoveContainer" containerID="87507c3962370a5a2b03a1577172811e456a74018323d5dfd6620d49eb20d8ce" Oct 31 01:15:22.276854 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:22.276754 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6"] Oct 31 01:15:22.280726 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:22.280699 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-85bp6"] Oct 31 01:15:23.934347 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:23.934312 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" path="/var/lib/kubelet/pods/49433dfd-fe62-48a1-90ff-e4fabd2d2c79/volumes" Oct 31 01:15:26.561708 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.561638 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:15:26.718064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718026 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-data\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718083 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-socket\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718173 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-envoy\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718212 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc1db1d-0066-42a5-8316-a66bda6e2408-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc1db1d-0066-42a5-8316-a66bda6e2408-istiod-ca-cert\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718281 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-credential-socket\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718322 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-data" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:26.718443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718356 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-token\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-token\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718399 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-podinfo\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718528 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-certs\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718561 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-kube-api-access-nbhz5\") pod \"6fc1db1d-0066-42a5-8316-a66bda6e2408\" (UID: \"6fc1db1d-0066-42a5-8316-a66bda6e2408\") " Oct 31 01:15:26.718656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718572 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-credential-socket" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:26.718838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.718811 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-certs" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:26.719053 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.719029 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-socket" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:26.719146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.719078 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc1db1d-0066-42a5-8316-a66bda6e2408-istiod-ca-cert" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:26.719146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.719125 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.719263 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.719147 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.719263 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.719162 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.721088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.721058 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-envoy" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:26.721340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.721301 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-podinfo" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:26.721442 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.721415 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-token" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:26.721872 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.721849 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-kube-api-access-nbhz5" pod "6fc1db1d-0066-42a5-8316-a66bda6e2408" (UID: "6fc1db1d-0066-42a5-8316-a66bda6e2408"). InnerVolumeSpecName "kube-api-access-nbhz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:26.819993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.819914 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.819993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.819943 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc1db1d-0066-42a5-8316-a66bda6e2408-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.819993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.819953 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.819993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.819963 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc1db1d-0066-42a5-8316-a66bda6e2408-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.819993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.819973 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/6fc1db1d-0066-42a5-8316-a66bda6e2408-kube-api-access-nbhz5\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:26.819993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:26.819983 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc1db1d-0066-42a5-8316-a66bda6e2408-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:27.156277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.156145 5890 generic.go:356] "Generic (PLEG): container finished" podID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerID="7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf" exitCode=0 Oct 31 01:15:27.156277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.156213 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" Oct 31 01:15:27.156277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.156232 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" event={"ID":"6fc1db1d-0066-42a5-8316-a66bda6e2408","Type":"ContainerDied","Data":"7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf"} Oct 31 01:15:27.156596 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.156299 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk" event={"ID":"6fc1db1d-0066-42a5-8316-a66bda6e2408","Type":"ContainerDied","Data":"fe18307ce185e4fb746ec22045921f5475ed5df0cda0348c04d9c0a791562101"} Oct 31 01:15:27.156596 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.156348 5890 scope.go:122] "RemoveContainer" containerID="7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf" Oct 31 01:15:27.168686 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.168669 5890 scope.go:122] "RemoveContainer" containerID="7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf" Oct 31 01:15:27.169073 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:27.169036 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf\": container with ID starting with 7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf not found: ID does not exist" containerID="7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf" Oct 31 01:15:27.169190 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.169078 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf"} err="failed to get container status \"7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf\": rpc error: code = NotFound desc = could not find container \"7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf\": container with ID starting with 7be4bef5a51be1b165be070dfc989c2d1209f07417f11ecb51669696c41fefdf not found: ID does not exist" Oct 31 01:15:27.316537 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.316482 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk"] Oct 31 01:15:27.364324 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.364287 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-dt5qk"] Oct 31 01:15:27.933861 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:27.933693 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" path="/var/lib/kubelet/pods/6fc1db1d-0066-42a5-8316-a66bda6e2408/volumes" Oct 31 01:15:37.018857 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:37.018817 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:15:47.377080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.377042 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk"] Oct 31 01:15:47.377730 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.377295 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" podUID="f4b10aef-764a-431f-8574-b1bb735caaa3" containerName="maas-api" containerID="cri-o://5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743" gracePeriod=30 Oct 31 01:15:47.663400 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.663367 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:15:47.768840 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.768805 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b10aef-764a-431f-8574-b1bb735caaa3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b10aef-764a-431f-8574-b1bb735caaa3-openshift-service-ca\") pod \"f4b10aef-764a-431f-8574-b1bb735caaa3\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " Oct 31 01:15:47.769058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.768927 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls\") pod \"f4b10aef-764a-431f-8574-b1bb735caaa3\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " Oct 31 01:15:47.769058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.769026 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b10aef-764a-431f-8574-b1bb735caaa3-kube-api-access-cmj75\" (UniqueName: \"kubernetes.io/projected/f4b10aef-764a-431f-8574-b1bb735caaa3-kube-api-access-cmj75\") pod \"f4b10aef-764a-431f-8574-b1bb735caaa3\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " Oct 31 01:15:47.769185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.769099 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-metrics-certs\") pod \"f4b10aef-764a-431f-8574-b1bb735caaa3\" (UID: \"f4b10aef-764a-431f-8574-b1bb735caaa3\") " Oct 31 01:15:47.770044 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.770015 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b10aef-764a-431f-8574-b1bb735caaa3-openshift-service-ca" pod "f4b10aef-764a-431f-8574-b1bb735caaa3" (UID: "f4b10aef-764a-431f-8574-b1bb735caaa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:47.776506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.776434 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls" pod "f4b10aef-764a-431f-8574-b1bb735caaa3" (UID: "f4b10aef-764a-431f-8574-b1bb735caaa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:47.779594 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.779540 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b10aef-764a-431f-8574-b1bb735caaa3-kube-api-access-cmj75" pod "f4b10aef-764a-431f-8574-b1bb735caaa3" (UID: "f4b10aef-764a-431f-8574-b1bb735caaa3"). InnerVolumeSpecName "kube-api-access-cmj75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:47.780068 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.780042 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-metrics-certs" pod "f4b10aef-764a-431f-8574-b1bb735caaa3" (UID: "f4b10aef-764a-431f-8574-b1bb735caaa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:47.816080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.816046 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2"] Oct 31 01:15:47.816328 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.816283 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" podUID="b067542f-0858-478b-b4b8-249b27d805a8" containerName="payload-processing" containerID="cri-o://2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029" gracePeriod=30 Oct 31 01:15:47.869021 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.868984 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2"] Oct 31 01:15:47.869236 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.869199 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" podUID="6ffee017-ab05-458b-bd42-92ef9eb9fd0d" containerName="payload-pre-processing" containerID="cri-o://be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64" gracePeriod=30 Oct 31 01:15:47.871718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.871692 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b10aef-764a-431f-8574-b1bb735caaa3-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:47.871718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.871718 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:47.871935 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.871729 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmj75\" (UniqueName: \"kubernetes.io/projected/f4b10aef-764a-431f-8574-b1bb735caaa3-kube-api-access-cmj75\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:47.871935 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:47.871738 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b10aef-764a-431f-8574-b1bb735caaa3-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:48.026757 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.025896 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" podUID="b067542f-0858-478b-b4b8-249b27d805a8" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.157:9004: connect: connection refused" Oct 31 01:15:48.136184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.136126 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:15:48.172760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.172733 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:15:48.276303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.276268 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-kube-api-access-hnq8z\" (UniqueName: \"kubernetes.io/projected/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-kube-api-access-hnq8z\") pod \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " Oct 31 01:15:48.276503 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.276390 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b067542f-0858-478b-b4b8-249b27d805a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b067542f-0858-478b-b4b8-249b27d805a8-plugins-config-volume\") pod \"b067542f-0858-478b-b4b8-249b27d805a8\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " Oct 31 01:15:48.276503 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.276440 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b067542f-0858-478b-b4b8-249b27d805a8-kube-api-access-qshvt\" (UniqueName: \"kubernetes.io/projected/b067542f-0858-478b-b4b8-249b27d805a8-kube-api-access-qshvt\") pod \"b067542f-0858-478b-b4b8-249b27d805a8\" (UID: \"b067542f-0858-478b-b4b8-249b27d805a8\") " Oct 31 01:15:48.276634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.276549 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-plugins-config-volume\") pod \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\" (UID: \"6ffee017-ab05-458b-bd42-92ef9eb9fd0d\") " Oct 31 01:15:48.277176 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.277142 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b067542f-0858-478b-b4b8-249b27d805a8-plugins-config-volume" pod "b067542f-0858-478b-b4b8-249b27d805a8" (UID: "b067542f-0858-478b-b4b8-249b27d805a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:48.277323 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.277187 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-plugins-config-volume" pod "6ffee017-ab05-458b-bd42-92ef9eb9fd0d" (UID: "6ffee017-ab05-458b-bd42-92ef9eb9fd0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:48.278724 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.278699 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-kube-api-access-hnq8z" pod "6ffee017-ab05-458b-bd42-92ef9eb9fd0d" (UID: "6ffee017-ab05-458b-bd42-92ef9eb9fd0d"). InnerVolumeSpecName "kube-api-access-hnq8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:48.279158 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.279137 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b067542f-0858-478b-b4b8-249b27d805a8-kube-api-access-qshvt" pod "b067542f-0858-478b-b4b8-249b27d805a8" (UID: "b067542f-0858-478b-b4b8-249b27d805a8"). InnerVolumeSpecName "kube-api-access-qshvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:48.327806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.316745 5890 generic.go:356] "Generic (PLEG): container finished" podID="6ffee017-ab05-458b-bd42-92ef9eb9fd0d" containerID="be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64" exitCode=0 Oct 31 01:15:48.327806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.316923 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" event={"ID":"6ffee017-ab05-458b-bd42-92ef9eb9fd0d","Type":"ContainerDied","Data":"be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64"} Oct 31 01:15:48.327806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.316979 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" event={"ID":"6ffee017-ab05-458b-bd42-92ef9eb9fd0d","Type":"ContainerDied","Data":"c404092a16b8070d3bfd87b2f11276fb8418bf3603e3d2f2c02db5e8fb09740c"} Oct 31 01:15:48.327806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.317010 5890 scope.go:122] "RemoveContainer" containerID="be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64" Oct 31 01:15:48.327806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.317167 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2" Oct 31 01:15:48.335472 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.335239 5890 generic.go:356] "Generic (PLEG): container finished" podID="f4b10aef-764a-431f-8574-b1bb735caaa3" containerID="5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743" exitCode=0 Oct 31 01:15:48.335848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.335508 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" Oct 31 01:15:48.335848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.335588 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" event={"ID":"f4b10aef-764a-431f-8574-b1bb735caaa3","Type":"ContainerDied","Data":"5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743"} Oct 31 01:15:48.336105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.335758 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk" event={"ID":"f4b10aef-764a-431f-8574-b1bb735caaa3","Type":"ContainerDied","Data":"3cca388c07cb90f914bf229da2fa5873f3c9c354833560640876969a44ef1f46"} Oct 31 01:15:48.341716 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.338581 5890 generic.go:356] "Generic (PLEG): container finished" podID="b067542f-0858-478b-b4b8-249b27d805a8" containerID="2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029" exitCode=0 Oct 31 01:15:48.341716 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.338687 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" event={"ID":"b067542f-0858-478b-b4b8-249b27d805a8","Type":"ContainerDied","Data":"2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029"} Oct 31 01:15:48.341716 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.338727 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" event={"ID":"b067542f-0858-478b-b4b8-249b27d805a8","Type":"ContainerDied","Data":"8beab89ccb5690db36ab635e4c5c41228b37a5ede64a1eaf1ec3e9f0b7bd5dda"} Oct 31 01:15:48.341716 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.338821 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2" Oct 31 01:15:48.378171 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.378134 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:48.378671 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.378650 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnq8z\" (UniqueName: \"kubernetes.io/projected/6ffee017-ab05-458b-bd42-92ef9eb9fd0d-kube-api-access-hnq8z\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:48.397158 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.397092 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b067542f-0858-478b-b4b8-249b27d805a8-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:48.397158 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.397131 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshvt\" (UniqueName: \"kubernetes.io/projected/b067542f-0858-478b-b4b8-249b27d805a8-kube-api-access-qshvt\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:48.410294 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.410253 5890 scope.go:122] "RemoveContainer" containerID="be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64" Oct 31 01:15:48.411181 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:48.410833 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64\": container with ID starting with be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64 not found: ID does not exist" containerID="be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64" Oct 31 01:15:48.411181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.410876 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64"} err="failed to get container status \"be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64\": rpc error: code = NotFound desc = could not find container \"be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64\": container with ID starting with be55640fad8eb07201d030bf82877c0ac86a8256c90ba5e0d45d4d493f5ede64 not found: ID does not exist" Oct 31 01:15:48.411181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.410908 5890 scope.go:122] "RemoveContainer" containerID="5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743" Oct 31 01:15:48.421453 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.421430 5890 scope.go:122] "RemoveContainer" containerID="5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743" Oct 31 01:15:48.421770 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:48.421747 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743\": container with ID starting with 5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743 not found: ID does not exist" containerID="5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743" Oct 31 01:15:48.421893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.421779 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743"} err="failed to get container status \"5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743\": rpc error: code = NotFound desc = could not find container \"5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743\": container with ID starting with 5890cf0de66192ab7b974a3935712795c87e1172d900696a73663774c4977743 not found: ID does not exist" Oct 31 01:15:48.421893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.421817 5890 scope.go:122] "RemoveContainer" containerID="2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029" Oct 31 01:15:48.431301 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.431281 5890 scope.go:122] "RemoveContainer" containerID="2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029" Oct 31 01:15:48.431573 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:15:48.431555 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029\": container with ID starting with 2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029 not found: ID does not exist" containerID="2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029" Oct 31 01:15:48.431625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.431582 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029"} err="failed to get container status \"2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029\": rpc error: code = NotFound desc = could not find container \"2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029\": container with ID starting with 2939a2e1e6db48b615d9f071fbe209a89b5cab47def9060759c4f06d3c509029 not found: ID does not exist" Oct 31 01:15:48.489890 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.489855 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2"] Oct 31 01:15:48.490940 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.490917 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-66e263-6c475c94c6-ffdw2"] Oct 31 01:15:48.595817 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.595766 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk"] Oct 31 01:15:48.600134 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.600109 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-66e263-7f7f8dc975-fkkkk"] Oct 31 01:15:48.704755 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.704721 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2"] Oct 31 01:15:48.710207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:48.710175 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-66e263-55994d8d9f-cb5r2"] Oct 31 01:15:49.934506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:49.934473 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffee017-ab05-458b-bd42-92ef9eb9fd0d" path="/var/lib/kubelet/pods/6ffee017-ab05-458b-bd42-92ef9eb9fd0d/volumes" Oct 31 01:15:49.934930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:49.934919 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b067542f-0858-478b-b4b8-249b27d805a8" path="/var/lib/kubelet/pods/b067542f-0858-478b-b4b8-249b27d805a8/volumes" Oct 31 01:15:49.935341 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:49.935326 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b10aef-764a-431f-8574-b1bb735caaa3" path="/var/lib/kubelet/pods/f4b10aef-764a-431f-8574-b1bb735caaa3/volumes" Oct 31 01:15:52.627215 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.627174 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt"] Oct 31 01:15:52.627963 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.627940 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.627963 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.627985 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b067542f-0858-478b-b4b8-249b27d805a8" containerName="payload-processing" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.627992 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b067542f-0858-478b-b4b8-249b27d805a8" containerName="payload-processing" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628008 5890 cpu_manager.go:414] "containerMap: removing container" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628017 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628028 5890 cpu_manager.go:414] "containerMap: removing container" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628035 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628049 5890 cpu_manager.go:414] "containerMap: removing container" podUID="f4b10aef-764a-431f-8574-b1bb735caaa3" containerName="maas-api" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628056 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b10aef-764a-431f-8574-b1bb735caaa3" containerName="maas-api" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628066 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:15:52.628088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628072 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628093 5890 cpu_manager.go:414] "containerMap: removing container" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628101 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628122 5890 cpu_manager.go:414] "containerMap: removing container" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628130 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628143 5890 cpu_manager.go:414] "containerMap: removing container" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628149 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628164 5890 cpu_manager.go:414] "containerMap: removing container" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628172 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628186 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628193 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628202 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6ffee017-ab05-458b-bd42-92ef9eb9fd0d" containerName="payload-pre-processing" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628210 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffee017-ab05-458b-bd42-92ef9eb9fd0d" containerName="payload-pre-processing" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628363 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628378 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628388 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628398 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc1db1d-0066-42a5-8316-a66bda6e2408" containerName="istio-proxy" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628413 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffee017-ab05-458b-bd42-92ef9eb9fd0d" containerName="payload-pre-processing" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628422 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628437 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b067542f-0858-478b-b4b8-249b27d805a8" containerName="payload-processing" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628446 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628457 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b10aef-764a-431f-8574-b1bb735caaa3" containerName="maas-api" Oct 31 01:15:52.628615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.628467 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:15:52.633860 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.633838 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.688092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.688059 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt"] Oct 31 01:15:52.731556 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.731515 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cb9a07\"/\"default-dockercfg-2s7zk\"" Oct 31 01:15:52.731556 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.731516 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cb9a07\"/\"openshift-service-ca.crt\"" Oct 31 01:15:52.732142 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.732121 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cb9a07\"/\"kube-root-ca.crt\"" Oct 31 01:15:52.732254 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.732123 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cb9a07\"/\"auth-test-model-cb9a07-kserve-self-signed-certs\"" Oct 31 01:15:52.755133 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.755104 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/302625c8-aefc-4514-8a60-4b802d25f826-tls-certs\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.755253 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.755161 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-home\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.755303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.755255 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-dshm\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.755350 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.755311 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-tmp-dir\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.755493 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.755472 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr958\" (UniqueName: \"kubernetes.io/projected/302625c8-aefc-4514-8a60-4b802d25f826-kube-api-access-sr958\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.755606 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.755589 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-model-cache\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857033 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.856998 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/302625c8-aefc-4514-8a60-4b802d25f826-tls-certs\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857067 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-home\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857094 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-dshm\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857141 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-tmp-dir\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857228 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr958\" (UniqueName: \"kubernetes.io/projected/302625c8-aefc-4514-8a60-4b802d25f826-kube-api-access-sr958\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857320 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-model-cache\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857688 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857654 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-home\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857718 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-model-cache\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.857834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.857816 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-tmp-dir\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.859974 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.859949 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-dshm\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.860223 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.860206 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/302625c8-aefc-4514-8a60-4b802d25f826-tls-certs\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.916438 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.916368 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr958\" (UniqueName: \"kubernetes.io/projected/302625c8-aefc-4514-8a60-4b802d25f826-kube-api-access-sr958\") pod \"auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:52.946602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:52.946569 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:53.325276 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:53.325216 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt"] Oct 31 01:15:53.376216 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:53.376187 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" event={"ID":"302625c8-aefc-4514-8a60-4b802d25f826","Type":"ContainerStarted","Data":"22a0ec3364d5fcdd86f0494aecfdb0b258b2ea30e4678951a70b427801393e1c"} Oct 31 01:15:54.383964 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:54.383925 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" event={"ID":"302625c8-aefc-4514-8a60-4b802d25f826","Type":"ContainerStarted","Data":"95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f"} Oct 31 01:15:54.498453 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:54.498396 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" podStartSLOduration=2.49838242 podStartE2EDuration="2.49838242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:54.496935684 +0000 UTC m=+2846.699186118" watchObservedRunningTime="2026-10-31 01:15:54.49838242 +0000 UTC m=+2846.700632853" Oct 31 01:15:54.947546 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:54.947507 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:15:56.720985 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:56.720946 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2"] Oct 31 01:15:56.721417 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:15:56.721197 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" containerID="cri-o://b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e" gracePeriod=30 Oct 31 01:16:02.016492 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.016462 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:16:02.072254 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072217 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-token\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-token\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072461 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072276 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-kube-api-access-krjbv\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-kube-api-access-krjbv\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072461 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072328 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-envoy\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072461 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072413 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-credential-socket\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072463 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-data\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072507 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-socket\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072547 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-certs\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072686 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c80169d-7e18-47d8-b7df-37f303823dcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80169d-7e18-47d8-b7df-37f303823dcf-istiod-ca-cert\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.072838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.072720 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-podinfo\") pod \"5c80169d-7e18-47d8-b7df-37f303823dcf\" (UID: \"5c80169d-7e18-47d8-b7df-37f303823dcf\") " Oct 31 01:16:02.073541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073122 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-certs" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:02.073541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073365 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-data" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:02.073541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073428 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-socket" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:02.073541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073492 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-credential-socket" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:02.073541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073501 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c80169d-7e18-47d8-b7df-37f303823dcf-istiod-ca-cert" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:02.073955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073661 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.073955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073680 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.073955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073697 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.073955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073711 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.073955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.073736 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c80169d-7e18-47d8-b7df-37f303823dcf-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.075114 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.075076 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-token" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:02.075227 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.075117 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-kube-api-access-krjbv" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "kube-api-access-krjbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:02.075328 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.075299 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-envoy" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:02.075870 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.075850 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-podinfo" pod "5c80169d-7e18-47d8-b7df-37f303823dcf" (UID: "5c80169d-7e18-47d8-b7df-37f303823dcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:16:02.175286 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.175178 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.175286 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.175216 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.175286 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.175231 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjbv\" (UniqueName: \"kubernetes.io/projected/5c80169d-7e18-47d8-b7df-37f303823dcf-kube-api-access-krjbv\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.175286 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.175247 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c80169d-7e18-47d8-b7df-37f303823dcf-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:02.440965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.440931 5890 generic.go:356] "Generic (PLEG): container finished" podID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerID="b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e" exitCode=0 Oct 31 01:16:02.441142 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.441001 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" Oct 31 01:16:02.441142 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.441021 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" event={"ID":"5c80169d-7e18-47d8-b7df-37f303823dcf","Type":"ContainerDied","Data":"b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e"} Oct 31 01:16:02.441142 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.441092 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2" event={"ID":"5c80169d-7e18-47d8-b7df-37f303823dcf","Type":"ContainerDied","Data":"d84456f3d50080fca4a851f6cf5a988498c6a701bdccf9c3f816eb26634d79ac"} Oct 31 01:16:02.441142 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.441134 5890 scope.go:122] "RemoveContainer" containerID="b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e" Oct 31 01:16:02.456963 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.456592 5890 scope.go:122] "RemoveContainer" containerID="3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65" Oct 31 01:16:02.468959 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.468932 5890 scope.go:122] "RemoveContainer" containerID="b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e" Oct 31 01:16:02.469304 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:16:02.469270 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e\": container with ID starting with b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e not found: ID does not exist" containerID="b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e" Oct 31 01:16:02.469423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.469315 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e"} err="failed to get container status \"b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e\": rpc error: code = NotFound desc = could not find container \"b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e\": container with ID starting with b372177373682e65baaa75b125933fee04de60b48db89381b8cba05f56109c5e not found: ID does not exist" Oct 31 01:16:02.469423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.469355 5890 scope.go:122] "RemoveContainer" containerID="3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65" Oct 31 01:16:02.469668 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:16:02.469639 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65\": container with ID starting with 3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65 not found: ID does not exist" containerID="3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65" Oct 31 01:16:02.469799 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.469677 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65"} err="failed to get container status \"3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65\": rpc error: code = NotFound desc = could not find container \"3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65\": container with ID starting with 3977a781e509e33232a4d9e347d2b936c15fcaeaa06b0eb7fb1c656468da2c65 not found: ID does not exist" Oct 31 01:16:02.601640 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.601599 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2"] Oct 31 01:16:02.651351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.651319 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-66e263-openshift-default-589754cbf-p52j2"] Oct 31 01:16:02.947519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.947484 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:16:02.965495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:02.965464 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:16:03.462870 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:03.462831 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:16:03.932839 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:03.932807 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" path="/var/lib/kubelet/pods/5c80169d-7e18-47d8-b7df-37f303823dcf/volumes" Oct 31 01:16:07.019442 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:16:07.019401 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:16:08.991427 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.991386 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt"] Oct 31 01:16:08.991999 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.991983 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:16:08.992067 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.992001 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:16:08.992067 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.992019 5890 cpu_manager.go:414] "containerMap: removing container" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" Oct 31 01:16:08.992067 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.992025 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" Oct 31 01:16:08.992163 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.992143 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" Oct 31 01:16:08.992163 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.992155 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="49433dfd-fe62-48a1-90ff-e4fabd2d2c79" containerName="cleanup" Oct 31 01:16:08.992163 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.992163 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4a8cc1-c449-42fa-8b0b-4c614b8935ae" containerName="cleanup" Oct 31 01:16:08.992262 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.992170 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f51c1e8-1899-4832-960d-ab98cbd588f8" containerName="cleanup" Oct 31 01:16:08.997042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:08.997024 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.060269 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.060224 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-certs\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.060633 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.060584 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-socket\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.060804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.060768 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-credential-socket\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.060986 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.060917 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-data\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.061179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.061149 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-envoy\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.061414 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.061391 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8075ec-c000-4339-a0ca-94a06f245742-istiod-ca-cert\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.061628 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.061603 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7rg5\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-kube-api-access-c7rg5\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.062167 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.062149 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8075ec-c000-4339-a0ca-94a06f245742-istio-podinfo\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.062530 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.062503 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-istio-token\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.068213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.068188 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt"] Oct 31 01:16:09.097630 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.097592 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-8ddd2c-openshift-default-dockercfg-5tw9f\"" Oct 31 01:16:09.164581 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164545 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-certs\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.164780 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164634 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-socket\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.164780 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164675 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-credential-socket\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.164780 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164705 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-data\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.164780 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164730 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-envoy\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.164780 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164758 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8075ec-c000-4339-a0ca-94a06f245742-istiod-ca-cert\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.165105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164885 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7rg5\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-kube-api-access-c7rg5\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.165105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.164920 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8075ec-c000-4339-a0ca-94a06f245742-istio-podinfo\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.165105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.165020 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-certs\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.165272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.165105 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-istio-token\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.165272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.165148 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-socket\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.165663 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.165635 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-data\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.165838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.165804 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8075ec-c000-4339-a0ca-94a06f245742-istiod-ca-cert\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.166048 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.166026 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-credential-socket\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.168042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.168014 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-envoy\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.168219 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.168202 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8075ec-c000-4339-a0ca-94a06f245742-istio-podinfo\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.238804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.235594 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7rg5\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-kube-api-access-c7rg5\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.243847 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.242234 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-istio-token\") pod \"e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.310924 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.310888 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:09.738611 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.738580 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt"] Oct 31 01:16:09.739687 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:16:09.739659 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8075ec_c000_4339_a0ca_94a06f245742.slice/crio-5e9cffde61f4422d3d9fc4c07474493c0c41564bd8678ec99cf337840626eab6 WatchSource:0}: Error finding container 5e9cffde61f4422d3d9fc4c07474493c0c41564bd8678ec99cf337840626eab6: Status 404 returned error can't find the container with id 5e9cffde61f4422d3d9fc4c07474493c0c41564bd8678ec99cf337840626eab6 Oct 31 01:16:09.742363 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.742325 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:16:09.742432 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.742399 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:16:09.742432 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:09.742427 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:16:10.510809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:10.510151 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" event={"ID":"3b8075ec-c000-4339-a0ca-94a06f245742","Type":"ContainerStarted","Data":"f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38"} Oct 31 01:16:10.510809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:10.510221 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" event={"ID":"3b8075ec-c000-4339-a0ca-94a06f245742","Type":"ContainerStarted","Data":"5e9cffde61f4422d3d9fc4c07474493c0c41564bd8678ec99cf337840626eab6"} Oct 31 01:16:10.679850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:10.679774 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" podStartSLOduration=2.679760993 podStartE2EDuration="2.679760993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:10.677876963 +0000 UTC m=+2862.880127396" watchObservedRunningTime="2026-10-31 01:16:10.679760993 +0000 UTC m=+2862.882011425" Oct 31 01:16:11.312080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:11.312043 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:11.319524 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:11.319489 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:11.520049 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:11.520018 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:11.521064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:11.521039 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:16:17.371021 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.370982 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h"] Oct 31 01:16:17.371552 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.371537 5890 cpu_manager.go:414] "containerMap: removing container" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" Oct 31 01:16:17.371603 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.371554 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" Oct 31 01:16:17.371676 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.371667 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c80169d-7e18-47d8-b7df-37f303823dcf" containerName="istio-proxy" Oct 31 01:16:17.375770 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.375746 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:17.429155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.429113 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h"] Oct 31 01:16:17.473805 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.473763 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8ddd2c\"" Oct 31 01:16:17.474030 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.473910 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8ddd2c-dockercfg-6s66x\"" Oct 31 01:16:17.477401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.477373 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15667028-1dda-411e-b4a4-1dc30d692dd5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:17.477540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.477492 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgvp\" (UniqueName: \"kubernetes.io/projected/15667028-1dda-411e-b4a4-1dc30d692dd5-kube-api-access-tvgvp\") pod \"payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:17.578985 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.578941 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15667028-1dda-411e-b4a4-1dc30d692dd5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:17.579199 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.579069 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgvp\" (UniqueName: \"kubernetes.io/projected/15667028-1dda-411e-b4a4-1dc30d692dd5-kube-api-access-tvgvp\") pod \"payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:17.579707 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.579685 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15667028-1dda-411e-b4a4-1dc30d692dd5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:17.643274 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.643203 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgvp\" (UniqueName: \"kubernetes.io/projected/15667028-1dda-411e-b4a4-1dc30d692dd5-kube-api-access-tvgvp\") pod \"payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:17.691188 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:17.691145 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:18.062287 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:18.062258 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h"] Oct 31 01:16:18.063576 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:16:18.063549 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15667028_1dda_411e_b4a4_1dc30d692dd5.slice/crio-438083898ad7cf0698a916b120460c1122f3d023cb3fa68ec58346fc637c97e2 WatchSource:0}: Error finding container 438083898ad7cf0698a916b120460c1122f3d023cb3fa68ec58346fc637c97e2: Status 404 returned error can't find the container with id 438083898ad7cf0698a916b120460c1122f3d023cb3fa68ec58346fc637c97e2 Oct 31 01:16:18.571710 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:18.571668 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" event={"ID":"15667028-1dda-411e-b4a4-1dc30d692dd5","Type":"ContainerStarted","Data":"6a915867d346d71aa6b7fd852556a81bf7a2e685ebeea039cbe3f5054c7dde26"} Oct 31 01:16:18.572481 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:18.571986 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:18.572481 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:18.572031 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" event={"ID":"15667028-1dda-411e-b4a4-1dc30d692dd5","Type":"ContainerStarted","Data":"438083898ad7cf0698a916b120460c1122f3d023cb3fa68ec58346fc637c97e2"} Oct 31 01:16:18.685311 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:18.685257 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" podStartSLOduration=1.68524271 podStartE2EDuration="1.68524271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:18.684177493 +0000 UTC m=+2870.886427927" watchObservedRunningTime="2026-10-31 01:16:18.68524271 +0000 UTC m=+2870.887493145" Oct 31 01:16:29.583592 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:16:29.583556 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:16:37.019314 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:16:37.019271 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:17:07.018150 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:17:07.018110 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:17:07.949706 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:07.949661 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd"] Oct 31 01:17:07.953876 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:07.953854 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.005509 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.005473 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9925a\"/\"openshift-service-ca.crt\"" Oct 31 01:17:08.005673 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.005585 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9925a\"/\"kube-root-ca.crt\"" Oct 31 01:17:08.053427 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.053396 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9925a\"/\"default-dockercfg-r44mx\"" Oct 31 01:17:08.053835 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.053540 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9925a\"/\"auth-test-model-f9925a-kserve-self-signed-certs\"" Oct 31 01:17:08.056578 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.056560 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd"] Oct 31 01:17:08.083656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.083630 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-tmp-dir\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.083796 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.083700 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvh57\" (UniqueName: \"kubernetes.io/projected/98ae480a-cc18-4970-9853-b8ec4191156a-kube-api-access-dvh57\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.083858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.083836 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae480a-cc18-4970-9853-b8ec4191156a-tls-certs\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.083900 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.083879 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-dshm\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.084025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.084012 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-home\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.084061 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.084052 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-model-cache\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.185343 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.185311 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae480a-cc18-4970-9853-b8ec4191156a-tls-certs\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.185585 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.185354 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-dshm\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.185585 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.185434 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-home\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.185585 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.185482 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-model-cache\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.185585 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.185518 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-tmp-dir\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.185834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.185586 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvh57\" (UniqueName: \"kubernetes.io/projected/98ae480a-cc18-4970-9853-b8ec4191156a-kube-api-access-dvh57\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.186013 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.185980 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-home\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.186156 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.186081 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-tmp-dir\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.186230 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.186182 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-model-cache\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.187927 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.187902 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-dshm\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.188363 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.188343 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae480a-cc18-4970-9853-b8ec4191156a-tls-certs\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.244908 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.244838 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvh57\" (UniqueName: \"kubernetes.io/projected/98ae480a-cc18-4970-9853-b8ec4191156a-kube-api-access-dvh57\") pod \"auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.265884 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.265855 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:08.640146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.640113 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd"] Oct 31 01:17:08.640824 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:17:08.640766 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ae480a_cc18_4970_9853_b8ec4191156a.slice/crio-99928504034d81c21634d91d96b5f175725e71f9f8781e2f4acef7cdb48002f1 WatchSource:0}: Error finding container 99928504034d81c21634d91d96b5f175725e71f9f8781e2f4acef7cdb48002f1: Status 404 returned error can't find the container with id 99928504034d81c21634d91d96b5f175725e71f9f8781e2f4acef7cdb48002f1 Oct 31 01:17:08.933359 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.933319 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" event={"ID":"98ae480a-cc18-4970-9853-b8ec4191156a","Type":"ContainerStarted","Data":"7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f"} Oct 31 01:17:08.933532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:08.933372 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" event={"ID":"98ae480a-cc18-4970-9853-b8ec4191156a","Type":"ContainerStarted","Data":"99928504034d81c21634d91d96b5f175725e71f9f8781e2f4acef7cdb48002f1"} Oct 31 01:17:09.046950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:09.046898 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" podStartSLOduration=2.046884614 podStartE2EDuration="2.046884614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:09.046437061 +0000 UTC m=+2921.248687493" watchObservedRunningTime="2026-10-31 01:17:09.046884614 +0000 UTC m=+2921.249135066" Oct 31 01:17:09.266187 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:09.266108 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:18.266349 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:18.266314 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:18.279278 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:18.279245 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:19.012522 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:19.012492 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:17:21.254693 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.254656 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt"] Oct 31 01:17:21.260177 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.260152 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.315883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.315853 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt"] Oct 31 01:17:21.344340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.344303 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tmp-dir\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.344539 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.344352 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-dshm\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.344539 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.344470 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-home\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.344539 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.344528 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-kube-api-access-h2stt\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.344731 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.344706 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tls-certs\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.344822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.344749 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-model-cache\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.360198 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.360170 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7f8596\"/\"openshift-service-ca.crt\"" Oct 31 01:17:21.360328 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.360290 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7f8596\"/\"kube-root-ca.crt\"" Oct 31 01:17:21.363008 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.362986 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7f8596\"/\"default-dockercfg-nhb9l\"" Oct 31 01:17:21.363008 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.363002 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7f8596\"/\"auth-test-model-7f8596-kserve-self-signed-certs\"" Oct 31 01:17:21.445890 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.445856 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tls-certs\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.445916 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-model-cache\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.446027 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tmp-dir\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.446058 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-dshm\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.446097 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-home\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446349 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.446153 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-kube-api-access-h2stt\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446448 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.446414 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-model-cache\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446514 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.446474 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tmp-dir\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.446741 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.446717 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-home\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.448776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.448753 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tls-certs\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.449058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.449039 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-dshm\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.506533 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.506465 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-kube-api-access-h2stt\") pod \"auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.578307 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.578263 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:21.954256 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:21.953619 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt"] Oct 31 01:17:22.025339 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:22.025305 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" event={"ID":"3b669c9b-3cb1-4aa5-accb-be9f8305fd85","Type":"ContainerStarted","Data":"60b66e7f85bbb5d7f2152f4aa8dca83215b2488facac7c512ad81d168cdcd4ff"} Oct 31 01:17:23.033951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:23.033918 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" event={"ID":"3b669c9b-3cb1-4aa5-accb-be9f8305fd85","Type":"ContainerStarted","Data":"c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716"} Oct 31 01:17:23.149617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:23.149563 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" podStartSLOduration=2.149549624 podStartE2EDuration="2.149549624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:23.1474068 +0000 UTC m=+2935.349657234" watchObservedRunningTime="2026-10-31 01:17:23.149549624 +0000 UTC m=+2935.351800056" Oct 31 01:17:23.578417 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:23.578382 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:31.579069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:31.579030 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:31.591797 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:31.591755 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:32.110798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:17:32.110760 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:17:37.020258 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:17:37.020219 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:18:07.019932 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:18:07.019891 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:18:07.359717 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:07.359615 5890 generic.go:356] "Generic (PLEG): container finished" podID="3b8075ec-c000-4339-a0ca-94a06f245742" containerID="f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38" exitCode=0 Oct 31 01:18:07.359717 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:07.359689 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" event={"ID":"3b8075ec-c000-4339-a0ca-94a06f245742","Type":"ContainerDied","Data":"f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38"} Oct 31 01:18:07.360104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:07.360087 5890 scope.go:122] "RemoveContainer" containerID="f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38" Oct 31 01:18:07.361522 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:07.361493 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:07.361605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:07.361562 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:07.361605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:07.361594 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:08.379831 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:08.379757 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" event={"ID":"3b8075ec-c000-4339-a0ca-94a06f245742","Type":"ContainerStarted","Data":"4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315"} Oct 31 01:18:09.311605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:09.311518 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:18:09.313133 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:09.313103 5890 patch_prober.go:29] interesting pod/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 01:18:09.313205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:09.313166 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 01:18:10.311214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:10.311175 5890 patch_prober.go:29] interesting pod/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 01:18:10.311617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:10.311233 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 01:18:11.312092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:11.312055 5890 patch_prober.go:29] interesting pod/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 01:18:11.312608 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:11.312108 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 01:18:11.520885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:11.520847 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:18:12.311988 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:12.311953 5890 patch_prober.go:29] interesting pod/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 01:18:12.312166 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:12.312007 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 01:18:13.311735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:13.311694 5890 patch_prober.go:29] interesting pod/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 01:18:13.311959 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:13.311763 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 01:18:14.316845 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:14.316815 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:18:14.317678 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:14.317652 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:18:28.931719 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:28.931691 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:18:29.048308 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:18:29.048280 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:18:37.020374 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:18:37.020333 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:19:07.022112 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:19:07.022066 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:19:07.896217 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:07.896179 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn"] Oct 31 01:19:07.896443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:07.896416 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" containerID="cri-o://77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4" gracePeriod=30 Oct 31 01:19:11.860038 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:11.859998 5890 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:19:11.860527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:11.860074 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:19:13.198894 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.198862 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:19:13.293599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.293568 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-envoy\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.293761 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.293672 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d4ef84-2c51-4a65-8a07-409da5190c45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d4ef84-2c51-4a65-8a07-409da5190c45-istiod-ca-cert\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.293837 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.293758 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-certs\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.293837 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.293828 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-kube-api-access-9pszr\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-kube-api-access-9pszr\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.293906 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.293870 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-podinfo\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.293941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.293916 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-credential-socket\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.294037 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294018 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-socket\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.294325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294251 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-data\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.294325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294039 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d4ef84-2c51-4a65-8a07-409da5190c45-istiod-ca-cert" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:13.294325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294063 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-certs" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:13.294325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294202 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-credential-socket" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:13.294325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294212 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-socket" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:13.294829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294359 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-token\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-token\") pod \"71d4ef84-2c51-4a65-8a07-409da5190c45\" (UID: \"71d4ef84-2c51-4a65-8a07-409da5190c45\") " Oct 31 01:19:13.294829 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294442 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-data" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:13.294938 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294919 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.294977 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294940 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.294977 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294955 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.294977 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294969 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d4ef84-2c51-4a65-8a07-409da5190c45-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.295076 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.294982 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.296089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.296063 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-envoy" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:13.296553 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.296525 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-kube-api-access-9pszr" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "kube-api-access-9pszr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:13.296657 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.296534 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-podinfo" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:19:13.296657 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.296617 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-token" pod "71d4ef84-2c51-4a65-8a07-409da5190c45" (UID: "71d4ef84-2c51-4a65-8a07-409da5190c45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:13.396044 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.396006 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pszr\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-kube-api-access-9pszr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.396044 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.396035 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.396044 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.396046 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.396044 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.396055 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d4ef84-2c51-4a65-8a07-409da5190c45-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:13.836299 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.836266 5890 generic.go:356] "Generic (PLEG): container finished" podID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerID="77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4" exitCode=0 Oct 31 01:19:13.836478 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.836313 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" Oct 31 01:19:13.836478 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.836344 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" event={"ID":"71d4ef84-2c51-4a65-8a07-409da5190c45","Type":"ContainerDied","Data":"77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4"} Oct 31 01:19:13.836478 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.836403 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn" event={"ID":"71d4ef84-2c51-4a65-8a07-409da5190c45","Type":"ContainerDied","Data":"be4e4be3f4a9e17982baa664320935cb1e7be15300f3fb295053c6d02bef8ab7"} Oct 31 01:19:13.836478 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.836434 5890 scope.go:122] "RemoveContainer" containerID="77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4" Oct 31 01:19:13.848318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.848301 5890 scope.go:122] "RemoveContainer" containerID="77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4" Oct 31 01:19:13.848602 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:19:13.848576 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4\": container with ID starting with 77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4 not found: ID does not exist" containerID="77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4" Oct 31 01:19:13.848671 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:13.848614 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4"} err="failed to get container status \"77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4\": rpc error: code = NotFound desc = could not find container \"77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4\": container with ID starting with 77efcf2885b1c9e937e23458c96a4bb27e0ce6338f24d3a3a3ff7ceee91a2ad4 not found: ID does not exist" Oct 31 01:19:14.054871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:14.054834 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn"] Oct 31 01:19:14.101242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:14.101170 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-6mskn"] Oct 31 01:19:15.935600 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:15.935556 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" path="/var/lib/kubelet/pods/71d4ef84-2c51-4a65-8a07-409da5190c45/volumes" Oct 31 01:19:36.864118 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:36.864077 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h"] Oct 31 01:19:37.019879 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:19:37.019838 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:19:41.864727 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:41.864683 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" podUID="15667028-1dda-411e-b4a4-1dc30d692dd5" containerName="payload-processing" containerID="cri-o://6a915867d346d71aa6b7fd852556a81bf7a2e685ebeea039cbe3f5054c7dde26" gracePeriod=25 Oct 31 01:19:42.066998 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.066966 5890 generic.go:356] "Generic (PLEG): container finished" podID="15667028-1dda-411e-b4a4-1dc30d692dd5" containerID="6a915867d346d71aa6b7fd852556a81bf7a2e685ebeea039cbe3f5054c7dde26" exitCode=0 Oct 31 01:19:42.067162 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.067102 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" event={"ID":"15667028-1dda-411e-b4a4-1dc30d692dd5","Type":"ContainerDied","Data":"6a915867d346d71aa6b7fd852556a81bf7a2e685ebeea039cbe3f5054c7dde26"} Oct 31 01:19:42.144841 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.144810 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:19:42.247182 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.247148 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15667028-1dda-411e-b4a4-1dc30d692dd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15667028-1dda-411e-b4a4-1dc30d692dd5-plugins-config-volume\") pod \"15667028-1dda-411e-b4a4-1dc30d692dd5\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " Oct 31 01:19:42.247379 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.247355 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15667028-1dda-411e-b4a4-1dc30d692dd5-kube-api-access-tvgvp\" (UniqueName: \"kubernetes.io/projected/15667028-1dda-411e-b4a4-1dc30d692dd5-kube-api-access-tvgvp\") pod \"15667028-1dda-411e-b4a4-1dc30d692dd5\" (UID: \"15667028-1dda-411e-b4a4-1dc30d692dd5\") " Oct 31 01:19:42.247626 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.247581 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15667028-1dda-411e-b4a4-1dc30d692dd5-plugins-config-volume" pod "15667028-1dda-411e-b4a4-1dc30d692dd5" (UID: "15667028-1dda-411e-b4a4-1dc30d692dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:42.247765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.247748 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15667028-1dda-411e-b4a4-1dc30d692dd5-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:42.249572 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.249540 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15667028-1dda-411e-b4a4-1dc30d692dd5-kube-api-access-tvgvp" pod "15667028-1dda-411e-b4a4-1dc30d692dd5" (UID: "15667028-1dda-411e-b4a4-1dc30d692dd5"). InnerVolumeSpecName "kube-api-access-tvgvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:42.349207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:42.349158 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvgvp\" (UniqueName: \"kubernetes.io/projected/15667028-1dda-411e-b4a4-1dc30d692dd5-kube-api-access-tvgvp\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:43.075763 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:43.075728 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" Oct 31 01:19:43.076273 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:43.075772 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h" event={"ID":"15667028-1dda-411e-b4a4-1dc30d692dd5","Type":"ContainerDied","Data":"438083898ad7cf0698a916b120460c1122f3d023cb3fa68ec58346fc637c97e2"} Oct 31 01:19:43.076273 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:43.075867 5890 scope.go:122] "RemoveContainer" containerID="6a915867d346d71aa6b7fd852556a81bf7a2e685ebeea039cbe3f5054c7dde26" Oct 31 01:19:43.235943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:43.235900 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h"] Oct 31 01:19:43.283803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:43.283753 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8ddd2c-6ffc4db587-f2g6h"] Oct 31 01:19:43.934186 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:19:43.934154 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15667028-1dda-411e-b4a4-1dc30d692dd5" path="/var/lib/kubelet/pods/15667028-1dda-411e-b4a4-1dc30d692dd5/volumes" Oct 31 01:20:07.019968 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:20:07.019924 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:20:29.768425 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:29.768385 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt"] Oct 31 01:20:29.768965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:29.768653 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" containerID="cri-o://4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315" gracePeriod=30 Oct 31 01:20:30.315079 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.315037 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z"] Oct 31 01:20:30.315704 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.315683 5890 cpu_manager.go:414] "containerMap: removing container" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" Oct 31 01:20:30.315858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.315705 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" Oct 31 01:20:30.315858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.315731 5890 cpu_manager.go:414] "containerMap: removing container" podUID="15667028-1dda-411e-b4a4-1dc30d692dd5" containerName="payload-processing" Oct 31 01:20:30.315858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.315739 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15667028-1dda-411e-b4a4-1dc30d692dd5" containerName="payload-processing" Oct 31 01:20:30.316032 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.315965 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d4ef84-2c51-4a65-8a07-409da5190c45" containerName="istio-proxy" Oct 31 01:20:30.316032 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.315985 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="15667028-1dda-411e-b4a4-1dc30d692dd5" containerName="payload-processing" Oct 31 01:20:30.320431 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.320411 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.382821 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.380684 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-eb68dd-dockercfg-fg82r\"" Oct 31 01:20:30.382821 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.380752 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-eb68dd\"" Oct 31 01:20:30.412123 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.412088 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z"] Oct 31 01:20:30.490238 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.490193 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksjs\" (UniqueName: \"kubernetes.io/projected/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-kube-api-access-cksjs\") pod \"payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.490417 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.490396 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.592379 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.592273 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.592746 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.592720 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksjs\" (UniqueName: \"kubernetes.io/projected/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-kube-api-access-cksjs\") pod \"payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.593365 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.593332 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.652595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.652557 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksjs\" (UniqueName: \"kubernetes.io/projected/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-kube-api-access-cksjs\") pod \"payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.934027 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.933996 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:30.974336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.974302 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd"] Oct 31 01:20:30.979749 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:30.979719 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.031231 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.031200 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c32349-openshift-default-dockercfg-st2v5\"" Oct 31 01:20:31.049566 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.049532 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd"] Oct 31 01:20:31.099394 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.099357 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4354ec0-c40c-4681-82b1-a02331ac929a-istiod-ca-cert\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.099556 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.099413 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-certs\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.099899 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.099865 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-socket\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.100005 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.099911 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-credential-socket\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.100005 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.099956 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-podinfo\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.100005 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.099980 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvkr\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-kube-api-access-9wvkr\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.100197 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.100056 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-envoy\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.100197 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.100096 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-data\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.100197 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.100159 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-token\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201335 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-podinfo\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201369 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvkr\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-kube-api-access-9wvkr\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201408 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-envoy\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201693 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201429 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-data\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201693 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201464 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-token\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201693 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201665 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4354ec0-c40c-4681-82b1-a02331ac929a-istiod-ca-cert\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201847 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201717 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-certs\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201847 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201780 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-socket\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201847 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201834 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-credential-socket\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.201997 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.201947 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-data\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.202165 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.202137 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-certs\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.202243 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.202183 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-credential-socket\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.202304 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.202241 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-socket\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.202547 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.202512 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4354ec0-c40c-4681-82b1-a02331ac929a-istiod-ca-cert\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.204388 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.204360 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-envoy\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.204509 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.204384 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-podinfo\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.260482 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.260453 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-token\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.261253 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.261236 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvkr\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-kube-api-access-9wvkr\") pod \"e2e-noten-c32349-openshift-default-848479bc65-mghjd\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.303270 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.303241 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:31.318880 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:20:31.318852 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276eb75d_7f2b_4451_bd5f_a7fc8ded55a7.slice/crio-d993dac955d0061465ff6a9b6441ca2b3708838c39febbcb9dbd81216371395c WatchSource:0}: Error finding container d993dac955d0061465ff6a9b6441ca2b3708838c39febbcb9dbd81216371395c: Status 404 returned error can't find the container with id d993dac955d0061465ff6a9b6441ca2b3708838c39febbcb9dbd81216371395c Oct 31 01:20:31.319160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.319137 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z"] Oct 31 01:20:31.437818 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.437759 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" event={"ID":"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7","Type":"ContainerStarted","Data":"d993dac955d0061465ff6a9b6441ca2b3708838c39febbcb9dbd81216371395c"} Oct 31 01:20:31.437964 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.437907 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:20:31.552834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.552722 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" podStartSLOduration=1.552708977 podStartE2EDuration="1.552708977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:31.550978515 +0000 UTC m=+3123.753228940" watchObservedRunningTime="2026-10-31 01:20:31.552708977 +0000 UTC m=+3123.754959442" Oct 31 01:20:31.682425 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.682390 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd"] Oct 31 01:20:31.682859 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:20:31.682832 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4354ec0_c40c_4681_82b1_a02331ac929a.slice/crio-f5592b8d9806a6741eccc45b21d5135a8bdc3782b2d46c4399cc9c512e46e7eb WatchSource:0}: Error finding container f5592b8d9806a6741eccc45b21d5135a8bdc3782b2d46c4399cc9c512e46e7eb: Status 404 returned error can't find the container with id f5592b8d9806a6741eccc45b21d5135a8bdc3782b2d46c4399cc9c512e46e7eb Oct 31 01:20:31.685489 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.685453 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:20:31.685600 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.685529 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:20:31.685600 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:31.685578 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:20:32.446206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:32.446169 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" event={"ID":"a4354ec0-c40c-4681-82b1-a02331ac929a","Type":"ContainerStarted","Data":"d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1"} Oct 31 01:20:32.446683 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:32.446230 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" event={"ID":"a4354ec0-c40c-4681-82b1-a02331ac929a","Type":"ContainerStarted","Data":"f5592b8d9806a6741eccc45b21d5135a8bdc3782b2d46c4399cc9c512e46e7eb"} Oct 31 01:20:32.447943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:32.447916 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" event={"ID":"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7","Type":"ContainerStarted","Data":"09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589"} Oct 31 01:20:32.614473 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:32.614414 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" podStartSLOduration=2.61439863 podStartE2EDuration="2.61439863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:32.609817861 +0000 UTC m=+3124.812068288" watchObservedRunningTime="2026-10-31 01:20:32.61439863 +0000 UTC m=+3124.816649063" Oct 31 01:20:33.304022 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:33.303981 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:33.311529 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:33.311497 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:33.455133 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:33.455094 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:33.455983 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:33.455962 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:20:35.037779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.037714 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:20:35.155567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.155530 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b8075ec-c000-4339-a0ca-94a06f245742-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8075ec-c000-4339-a0ca-94a06f245742-istio-podinfo\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.155732 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.155589 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-socket\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.155732 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.155627 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-istio-token\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-istio-token\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.155732 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.155706 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-certs\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.155961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.155892 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-socket" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:35.155961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.155898 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-envoy\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.156126 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156000 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-kube-api-access-c7rg5\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-kube-api-access-c7rg5\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.156126 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156051 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-data\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.156126 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156093 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-credential-socket\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.156126 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156120 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8075ec-c000-4339-a0ca-94a06f245742-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8075ec-c000-4339-a0ca-94a06f245742-istiod-ca-cert\") pod \"3b8075ec-c000-4339-a0ca-94a06f245742\" (UID: \"3b8075ec-c000-4339-a0ca-94a06f245742\") " Oct 31 01:20:35.156341 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156237 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-certs" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:35.156407 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156386 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-credential-socket" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:35.156893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156531 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-data" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:35.156893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156761 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.156893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156748 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8075ec-c000-4339-a0ca-94a06f245742-istiod-ca-cert" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:35.156893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156807 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.156893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156824 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.156893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.156838 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.158226 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.158202 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b8075ec-c000-4339-a0ca-94a06f245742-istio-podinfo" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:20:35.158564 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.158544 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-envoy" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:35.158666 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.158592 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-istio-token" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:35.158731 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.158687 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-kube-api-access-c7rg5" pod "3b8075ec-c000-4339-a0ca-94a06f245742" (UID: "3b8075ec-c000-4339-a0ca-94a06f245742"). InnerVolumeSpecName "kube-api-access-c7rg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:35.257905 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.257864 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8075ec-c000-4339-a0ca-94a06f245742-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.257905 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.257905 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7rg5\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-kube-api-access-c7rg5\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.258096 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.257922 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8075ec-c000-4339-a0ca-94a06f245742-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.258096 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.257936 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8075ec-c000-4339-a0ca-94a06f245742-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.258096 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.257950 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8075ec-c000-4339-a0ca-94a06f245742-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:35.471851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.471814 5890 generic.go:356] "Generic (PLEG): container finished" podID="3b8075ec-c000-4339-a0ca-94a06f245742" containerID="4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315" exitCode=0 Oct 31 01:20:35.472066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.471876 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" Oct 31 01:20:35.472066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.471902 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" event={"ID":"3b8075ec-c000-4339-a0ca-94a06f245742","Type":"ContainerDied","Data":"4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315"} Oct 31 01:20:35.472066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.471965 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt" event={"ID":"3b8075ec-c000-4339-a0ca-94a06f245742","Type":"ContainerDied","Data":"5e9cffde61f4422d3d9fc4c07474493c0c41564bd8678ec99cf337840626eab6"} Oct 31 01:20:35.472066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.471990 5890 scope.go:122] "RemoveContainer" containerID="4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315" Oct 31 01:20:35.483879 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.483772 5890 scope.go:122] "RemoveContainer" containerID="f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38" Oct 31 01:20:35.494585 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.494566 5890 scope.go:122] "RemoveContainer" containerID="4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315" Oct 31 01:20:35.494916 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:20:35.494888 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315\": container with ID starting with 4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315 not found: ID does not exist" containerID="4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315" Oct 31 01:20:35.494998 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.494921 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315"} err="failed to get container status \"4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315\": rpc error: code = NotFound desc = could not find container \"4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315\": container with ID starting with 4e49582aac62b79f389d7ee002fbb150272e3b9a0a15028036c308f5719f6315 not found: ID does not exist" Oct 31 01:20:35.494998 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.494944 5890 scope.go:122] "RemoveContainer" containerID="f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38" Oct 31 01:20:35.495209 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:20:35.495190 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38\": container with ID starting with f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38 not found: ID does not exist" containerID="f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38" Oct 31 01:20:35.495246 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.495215 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38"} err="failed to get container status \"f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38\": rpc error: code = NotFound desc = could not find container \"f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38\": container with ID starting with f3a3a2409d3711ade4a1bba1650fec22b489b4ede26b3b2d86d5c9590ad5bf38 not found: ID does not exist" Oct 31 01:20:35.635825 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.635771 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt"] Oct 31 01:20:35.683139 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.683097 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-8ddd2c-openshift-default-855cd88c9c-kk6zt"] Oct 31 01:20:35.934126 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:35.934092 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" path="/var/lib/kubelet/pods/3b8075ec-c000-4339-a0ca-94a06f245742/volumes" Oct 31 01:20:37.019182 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:20:37.019142 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:20:43.458578 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:20:43.458538 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:21:07.019514 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:21:07.019478 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:21:32.580727 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.580684 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p"] Oct 31 01:21:32.581632 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.581609 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" Oct 31 01:21:32.581682 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.581636 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" Oct 31 01:21:32.581899 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.581881 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" Oct 31 01:21:32.581941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.581912 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" Oct 31 01:21:32.586481 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.586461 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.638074 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.637103 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-eb68dd-serving-cert\"" Oct 31 01:21:32.678469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.678432 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmm5\" (UniqueName: \"kubernetes.io/projected/33658ba2-9a0f-4edd-9559-7023b7841b73-kube-api-access-pzmm5\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.678641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.678575 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-metrics-certs\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.678641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.678612 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33658ba2-9a0f-4edd-9559-7023b7841b73-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.678778 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.678765 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.689763 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.689726 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p"] Oct 31 01:21:32.780413 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.780373 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.780634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.780492 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmm5\" (UniqueName: \"kubernetes.io/projected/33658ba2-9a0f-4edd-9559-7023b7841b73-kube-api-access-pzmm5\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.780634 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:21:32.780528 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-serving-cert: secret "maas-api-e2e-extmod-a-eb68dd-serving-cert" not found Oct 31 01:21:32.780634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.780556 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-metrics-certs\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.780634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.780587 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33658ba2-9a0f-4edd-9559-7023b7841b73-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.780634 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:21:32.780616 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls podName:33658ba2-9a0f-4edd-9559-7023b7841b73 nodeName:}" failed. No retries permitted until 2026-10-31 01:21:33.280592891 +0000 UTC m=+3185.482843323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls") pod "maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" (UID: "33658ba2-9a0f-4edd-9559-7023b7841b73") : secret "maas-api-e2e-extmod-a-eb68dd-serving-cert" not found Oct 31 01:21:32.781347 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.781316 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33658ba2-9a0f-4edd-9559-7023b7841b73-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.783464 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.783392 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-metrics-certs\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:32.840773 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:32.840698 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmm5\" (UniqueName: \"kubernetes.io/projected/33658ba2-9a0f-4edd-9559-7023b7841b73-kube-api-access-pzmm5\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:33.287545 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:33.287505 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:33.290230 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:33.290201 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:33.502125 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:33.502089 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:33.881878 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:33.881843 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p"] Oct 31 01:21:33.885139 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:33.885116 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:21:34.897277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:34.897241 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" event={"ID":"33658ba2-9a0f-4edd-9559-7023b7841b73","Type":"ContainerStarted","Data":"749a31c88cddf70ca09c01814d693c91a7a0569035afda275d0cabd99c43163f"} Oct 31 01:21:35.906324 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:35.906276 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" event={"ID":"33658ba2-9a0f-4edd-9559-7023b7841b73","Type":"ContainerStarted","Data":"d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5"} Oct 31 01:21:35.906766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:35.906372 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:36.022117 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:36.022062 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" podStartSLOduration=3.075220272 podStartE2EDuration="4.022048545s" totalImagesPullingTime="946.828273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:36.019913321 +0000 UTC m=+3188.222163754" watchObservedRunningTime="2026-10-31 01:21:36.022048545 +0000 UTC m=+3188.224298955" Oct 31 01:21:37.023269 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:21:37.023225 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:21:41.919377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:41.919346 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:21:45.532882 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:45.532847 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd"] Oct 31 01:21:45.533320 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:45.533043 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" podUID="a4354ec0-c40c-4681-82b1-a02331ac929a" containerName="istio-proxy" containerID="cri-o://d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1" gracePeriod=30 Oct 31 01:21:48.457579 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:48.457539 5890 patch_prober.go:29] interesting pod/e2e-noten-c32349-openshift-default-848479bc65-mghjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:21:48.458010 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:48.457607 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" podUID="a4354ec0-c40c-4681-82b1-a02331ac929a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:21:50.829652 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:50.829620 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:21:51.003104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003063 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-podinfo\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003119 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-kube-api-access-9wvkr\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-kube-api-access-9wvkr\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003373 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003157 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-certs\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003373 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003300 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-credential-socket\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003488 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003400 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-socket\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003488 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003454 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-data\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003488 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003455 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-certs" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.003636 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003551 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-token\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-token\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003636 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003587 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4354ec0-c40c-4681-82b1-a02331ac929a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4354ec0-c40c-4681-82b1-a02331ac929a-istiod-ca-cert\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003641 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-envoy\") pod \"a4354ec0-c40c-4681-82b1-a02331ac929a\" (UID: \"a4354ec0-c40c-4681-82b1-a02331ac929a\") " Oct 31 01:21:51.003740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003661 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-credential-socket" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.003740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.003704 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-socket" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.004045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.004017 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4354ec0-c40c-4681-82b1-a02331ac929a-istiod-ca-cert" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:51.004329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.004306 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.004415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.004336 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.004415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.004352 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.004415 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.004367 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4354ec0-c40c-4681-82b1-a02331ac929a-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.004580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.004487 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-data" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.006003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.005976 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-kube-api-access-9wvkr" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "kube-api-access-9wvkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:51.006294 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.006265 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-token" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:51.006533 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.006512 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-podinfo" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:21:51.006702 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.006684 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-envoy" pod "a4354ec0-c40c-4681-82b1-a02331ac929a" (UID: "a4354ec0-c40c-4681-82b1-a02331ac929a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.032912 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.032885 5890 generic.go:356] "Generic (PLEG): container finished" podID="a4354ec0-c40c-4681-82b1-a02331ac929a" containerID="d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1" exitCode=0 Oct 31 01:21:51.033041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.032953 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" Oct 31 01:21:51.033041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.032969 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" event={"ID":"a4354ec0-c40c-4681-82b1-a02331ac929a","Type":"ContainerDied","Data":"d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1"} Oct 31 01:21:51.033041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.033037 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd" event={"ID":"a4354ec0-c40c-4681-82b1-a02331ac929a","Type":"ContainerDied","Data":"f5592b8d9806a6741eccc45b21d5135a8bdc3782b2d46c4399cc9c512e46e7eb"} Oct 31 01:21:51.033263 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.033069 5890 scope.go:122] "RemoveContainer" containerID="d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1" Oct 31 01:21:51.044496 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.044469 5890 scope.go:122] "RemoveContainer" containerID="d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1" Oct 31 01:21:51.044767 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:21:51.044749 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1\": container with ID starting with d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1 not found: ID does not exist" containerID="d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1" Oct 31 01:21:51.044896 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.044774 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1"} err="failed to get container status \"d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1\": rpc error: code = NotFound desc = could not find container \"d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1\": container with ID starting with d0113fe920693e4ff028134a0401079b659be093d6ad9e7b457165ae34d7cdb1 not found: ID does not exist" Oct 31 01:21:51.106136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.106055 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvkr\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-kube-api-access-9wvkr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.106136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.106085 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.106136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.106095 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.106136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.106104 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.106136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.106113 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4354ec0-c40c-4681-82b1-a02331ac929a-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.194067 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.194030 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd"] Oct 31 01:21:51.198894 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.198862 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c32349-openshift-default-848479bc65-mghjd"] Oct 31 01:21:51.934265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:51.934232 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4354ec0-c40c-4681-82b1-a02331ac929a" path="/var/lib/kubelet/pods/a4354ec0-c40c-4681-82b1-a02331ac929a/volumes" Oct 31 01:21:52.654459 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.654425 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm"] Oct 31 01:21:52.655333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.655312 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" Oct 31 01:21:52.655420 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.655336 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8075ec-c000-4339-a0ca-94a06f245742" containerName="istio-proxy" Oct 31 01:21:52.655420 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.655358 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a4354ec0-c40c-4681-82b1-a02331ac929a" containerName="istio-proxy" Oct 31 01:21:52.655420 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.655367 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4354ec0-c40c-4681-82b1-a02331ac929a" containerName="istio-proxy" Oct 31 01:21:52.655569 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.655552 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4354ec0-c40c-4681-82b1-a02331ac929a" containerName="istio-proxy" Oct 31 01:21:52.659913 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.659892 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.750214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.750175 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm"] Oct 31 01:21:52.758407 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.758376 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3aca3f-openshift-default-dockercfg-bshtj\"" Oct 31 01:21:52.826612 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.826572 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-istio-token\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.826804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.826644 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5016469-de22-402b-b73a-471dab674b25-istio-podinfo\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.826804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.826753 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-envoy\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.826895 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.826827 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-certs\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.826934 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.826887 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5016469-de22-402b-b73a-471dab674b25-istiod-ca-cert\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.826970 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.826944 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-credential-socket\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.827019 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.827006 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-socket\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.827056 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.827035 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jts9k\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-kube-api-access-jts9k\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.827105 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.827062 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-data\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928065 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928029 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-envoy\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928115 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-certs\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928154 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5016469-de22-402b-b73a-471dab674b25-istiod-ca-cert\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928249 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-credential-socket\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928316 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-socket\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928378 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jts9k\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-kube-api-access-jts9k\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928506 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928410 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-data\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928674 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928568 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-istio-token\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928674 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928619 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5016469-de22-402b-b73a-471dab674b25-istio-podinfo\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928674 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928652 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-credential-socket\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.928855 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.928731 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-socket\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.929095 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.929066 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5016469-de22-402b-b73a-471dab674b25-istiod-ca-cert\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.929204 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.929181 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-certs\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.929259 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.929246 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-data\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.930387 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.930367 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-envoy\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.930978 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.930958 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5016469-de22-402b-b73a-471dab674b25-istio-podinfo\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.991737 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.991706 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-istio-token\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:52.992155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:52.992120 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jts9k\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-kube-api-access-jts9k\") pod \"e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:53.273819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:53.273708 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:53.698620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:53.698589 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm"] Oct 31 01:21:53.699601 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:21:53.699573 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5016469_de22_402b_b73a_471dab674b25.slice/crio-bbdf08b2aa562b55b4c4e4028a1c78db6243f2a2a741af890758542fdf319159 WatchSource:0}: Error finding container bbdf08b2aa562b55b4c4e4028a1c78db6243f2a2a741af890758542fdf319159: Status 404 returned error can't find the container with id bbdf08b2aa562b55b4c4e4028a1c78db6243f2a2a741af890758542fdf319159 Oct 31 01:21:53.702155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:53.702120 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:21:53.702244 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:53.702188 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:21:53.702244 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:53.702217 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:21:54.060611 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:54.060497 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" event={"ID":"a5016469-de22-402b-b73a-471dab674b25","Type":"ContainerStarted","Data":"be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f"} Oct 31 01:21:54.060611 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:54.060578 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" event={"ID":"a5016469-de22-402b-b73a-471dab674b25","Type":"ContainerStarted","Data":"bbdf08b2aa562b55b4c4e4028a1c78db6243f2a2a741af890758542fdf319159"} Oct 31 01:21:54.179191 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:54.179137 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podStartSLOduration=2.1791244 podStartE2EDuration="2.1791244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:54.17778035 +0000 UTC m=+3206.380030783" watchObservedRunningTime="2026-10-31 01:21:54.1791244 +0000 UTC m=+3206.381374833" Oct 31 01:21:54.274743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:54.274708 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:54.281015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:54.280981 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:55.069073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:55.069042 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:21:55.070291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:21:55.070266 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:22:00.447923 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:00.447883 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt"] Oct 31 01:22:07.021966 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:22:07.021927 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:22:15.469699 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.469654 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" podUID="302625c8-aefc-4514-8a60-4b802d25f826" containerName="main" containerID="cri-o://95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f" gracePeriod=45 Oct 31 01:22:15.737291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.737231 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:22:15.875400 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.875367 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302625c8-aefc-4514-8a60-4b802d25f826-kube-api-access-sr958\" (UniqueName: \"kubernetes.io/projected/302625c8-aefc-4514-8a60-4b802d25f826-kube-api-access-sr958\") pod \"302625c8-aefc-4514-8a60-4b802d25f826\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " Oct 31 01:22:15.875593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.875471 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-model-cache\") pod \"302625c8-aefc-4514-8a60-4b802d25f826\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " Oct 31 01:22:15.875593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.875584 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-home\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-home\") pod \"302625c8-aefc-4514-8a60-4b802d25f826\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " Oct 31 01:22:15.875718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.875625 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302625c8-aefc-4514-8a60-4b802d25f826-tls-certs\" (UniqueName: \"kubernetes.io/secret/302625c8-aefc-4514-8a60-4b802d25f826-tls-certs\") pod \"302625c8-aefc-4514-8a60-4b802d25f826\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " Oct 31 01:22:15.875834 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.875818 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-tmp-dir\") pod \"302625c8-aefc-4514-8a60-4b802d25f826\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " Oct 31 01:22:15.875908 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.875862 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-dshm\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-dshm\") pod \"302625c8-aefc-4514-8a60-4b802d25f826\" (UID: \"302625c8-aefc-4514-8a60-4b802d25f826\") " Oct 31 01:22:15.876143 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.876114 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-tmp-dir" pod "302625c8-aefc-4514-8a60-4b802d25f826" (UID: "302625c8-aefc-4514-8a60-4b802d25f826"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.876216 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.876132 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-home" pod "302625c8-aefc-4514-8a60-4b802d25f826" (UID: "302625c8-aefc-4514-8a60-4b802d25f826"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.876407 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.876382 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.876510 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.876412 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.876769 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.876737 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-model-cache" pod "302625c8-aefc-4514-8a60-4b802d25f826" (UID: "302625c8-aefc-4514-8a60-4b802d25f826"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.879092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.878910 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-dshm" pod "302625c8-aefc-4514-8a60-4b802d25f826" (UID: "302625c8-aefc-4514-8a60-4b802d25f826"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.879092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.879005 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302625c8-aefc-4514-8a60-4b802d25f826-tls-certs" pod "302625c8-aefc-4514-8a60-4b802d25f826" (UID: "302625c8-aefc-4514-8a60-4b802d25f826"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:15.879092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.879057 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302625c8-aefc-4514-8a60-4b802d25f826-kube-api-access-sr958" pod "302625c8-aefc-4514-8a60-4b802d25f826" (UID: "302625c8-aefc-4514-8a60-4b802d25f826"). InnerVolumeSpecName "kube-api-access-sr958". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:15.977686 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.977636 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.977686 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.977669 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/302625c8-aefc-4514-8a60-4b802d25f826-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.977686 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.977683 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/302625c8-aefc-4514-8a60-4b802d25f826-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.977970 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:15.977698 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr958\" (UniqueName: \"kubernetes.io/projected/302625c8-aefc-4514-8a60-4b802d25f826-kube-api-access-sr958\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:16.223345 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.223310 5890 generic.go:356] "Generic (PLEG): container finished" podID="302625c8-aefc-4514-8a60-4b802d25f826" containerID="95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f" exitCode=0 Oct 31 01:22:16.223549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.223354 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" Oct 31 01:22:16.223549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.223396 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" event={"ID":"302625c8-aefc-4514-8a60-4b802d25f826","Type":"ContainerDied","Data":"95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f"} Oct 31 01:22:16.223549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.223438 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt" event={"ID":"302625c8-aefc-4514-8a60-4b802d25f826","Type":"ContainerDied","Data":"22a0ec3364d5fcdd86f0494aecfdb0b258b2ea30e4678951a70b427801393e1c"} Oct 31 01:22:16.223549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.223455 5890 scope.go:122] "RemoveContainer" containerID="95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f" Oct 31 01:22:16.238077 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.238056 5890 scope.go:122] "RemoveContainer" containerID="95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f" Oct 31 01:22:16.238375 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:22:16.238353 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f\": container with ID starting with 95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f not found: ID does not exist" containerID="95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f" Oct 31 01:22:16.238436 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.238386 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f"} err="failed to get container status \"95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f\": rpc error: code = NotFound desc = could not find container \"95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f\": container with ID starting with 95302e019891f4ac05d3c9dc1ef50091b3ca8f8b2be5487fbfe9dd23db589d3f not found: ID does not exist" Oct 31 01:22:16.383077 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.383035 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt"] Oct 31 01:22:16.387929 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:16.387899 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cb9a07/auth-test-model-cb9a07-kserve-6fdd45dd4f-cbvjt"] Oct 31 01:22:17.933288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:17.933259 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302625c8-aefc-4514-8a60-4b802d25f826" path="/var/lib/kubelet/pods/302625c8-aefc-4514-8a60-4b802d25f826/volumes" Oct 31 01:22:33.057753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.057698 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr"] Oct 31 01:22:33.058332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.058313 5890 cpu_manager.go:414] "containerMap: removing container" podUID="302625c8-aefc-4514-8a60-4b802d25f826" containerName="main" Oct 31 01:22:33.058391 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.058333 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302625c8-aefc-4514-8a60-4b802d25f826" containerName="main" Oct 31 01:22:33.058475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.058466 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="302625c8-aefc-4514-8a60-4b802d25f826" containerName="main" Oct 31 01:22:33.061589 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.061568 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.077747 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.077707 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh47n\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-kube-api-access-jh47n\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.077910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.077765 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-socket\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.077989 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.077947 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-token\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.078043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.077988 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-certs\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.078043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.078016 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-envoy\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.078156 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.078096 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-credential-socket\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.078156 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.078146 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-data\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.078259 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.078183 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-podinfo\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.078259 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.078236 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a53953-ecea-4303-8be1-99b6f93bc3e9-istiod-ca-cert\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.118514 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.118479 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-6b5091-openshift-default-dockercfg-zrqdn\"" Oct 31 01:22:33.153650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.153619 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr"] Oct 31 01:22:33.179036 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179001 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-credential-socket\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179230 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179048 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-data\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179230 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179075 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-podinfo\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179230 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179130 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a53953-ecea-4303-8be1-99b6f93bc3e9-istiod-ca-cert\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179465 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179235 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh47n\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-kube-api-access-jh47n\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179465 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179268 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-socket\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179465 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179331 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-token\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179465 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179441 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-certs\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179487 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-envoy\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179492 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-credential-socket\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179687 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-socket\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179898 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179725 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-data\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.179898 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.179889 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-certs\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.180398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.180374 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a53953-ecea-4303-8be1-99b6f93bc3e9-istiod-ca-cert\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.181535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.181515 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-envoy\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.181695 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.181677 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-podinfo\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.241233 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.241202 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-token\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.241233 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.241229 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh47n\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-kube-api-access-jh47n\") pod \"e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.376232 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.376154 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:33.801050 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.801014 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr"] Oct 31 01:22:33.802775 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:22:33.802742 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a53953_ecea_4303_8be1_99b6f93bc3e9.slice/crio-785cd89edafb4bc718973f6fc4d3157f14000f25d1562645829100232d615efc WatchSource:0}: Error finding container 785cd89edafb4bc718973f6fc4d3157f14000f25d1562645829100232d615efc: Status 404 returned error can't find the container with id 785cd89edafb4bc718973f6fc4d3157f14000f25d1562645829100232d615efc Oct 31 01:22:33.804705 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.804671 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:33.804774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.804736 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:33.804774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:33.804768 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:34.345507 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:34.345469 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" event={"ID":"31a53953-ecea-4303-8be1-99b6f93bc3e9","Type":"ContainerStarted","Data":"446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b"} Oct 31 01:22:34.346030 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:34.345523 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" event={"ID":"31a53953-ecea-4303-8be1-99b6f93bc3e9","Type":"ContainerStarted","Data":"785cd89edafb4bc718973f6fc4d3157f14000f25d1562645829100232d615efc"} Oct 31 01:22:34.376266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:34.376239 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:34.382320 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:34.382298 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:34.464887 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:34.464836 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podStartSLOduration=1.464821223 podStartE2EDuration="1.464821223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:34.462840123 +0000 UTC m=+3246.665090553" watchObservedRunningTime="2026-10-31 01:22:34.464821223 +0000 UTC m=+3246.667071653" Oct 31 01:22:35.353121 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:35.353077 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:35.354049 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:35.354028 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:22:37.020093 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:22:37.020057 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:22:46.107488 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.107454 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd"] Oct 31 01:22:46.435799 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.435749 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k"] Oct 31 01:22:46.442627 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.442132 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.486014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.485982 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k"] Oct 31 01:22:46.493417 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.493386 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6b5091-dockercfg-gg5pn\"" Oct 31 01:22:46.526373 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.526341 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6b5091\"" Oct 31 01:22:46.553815 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.553771 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/807f2910-69bb-4094-9027-5bf693b64fb3-plugins-config-volume\") pod \"payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.553973 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.553949 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xssrl\" (UniqueName: \"kubernetes.io/projected/807f2910-69bb-4094-9027-5bf693b64fb3-kube-api-access-xssrl\") pod \"payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.657807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.657550 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssrl\" (UniqueName: \"kubernetes.io/projected/807f2910-69bb-4094-9027-5bf693b64fb3-kube-api-access-xssrl\") pod \"payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.658028 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.657885 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/807f2910-69bb-4094-9027-5bf693b64fb3-plugins-config-volume\") pod \"payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.663806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.658894 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/807f2910-69bb-4094-9027-5bf693b64fb3-plugins-config-volume\") pod \"payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.724617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.724528 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xssrl\" (UniqueName: \"kubernetes.io/projected/807f2910-69bb-4094-9027-5bf693b64fb3-kube-api-access-xssrl\") pod \"payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.765213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.765185 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:46.998655 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:46.998556 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt"] Oct 31 01:22:47.173460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:47.173430 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k"] Oct 31 01:22:47.175061 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:22:47.175029 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807f2910_69bb_4094_9027_5bf693b64fb3.slice/crio-c5407dd10eebe110af6f7d662b81ccc053cbdad2e2a16db5a47eac8ded1b92f4 WatchSource:0}: Error finding container c5407dd10eebe110af6f7d662b81ccc053cbdad2e2a16db5a47eac8ded1b92f4: Status 404 returned error can't find the container with id c5407dd10eebe110af6f7d662b81ccc053cbdad2e2a16db5a47eac8ded1b92f4 Oct 31 01:22:47.449904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:47.449852 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" event={"ID":"807f2910-69bb-4094-9027-5bf693b64fb3","Type":"ContainerStarted","Data":"8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a"} Oct 31 01:22:47.449904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:47.449909 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" event={"ID":"807f2910-69bb-4094-9027-5bf693b64fb3","Type":"ContainerStarted","Data":"c5407dd10eebe110af6f7d662b81ccc053cbdad2e2a16db5a47eac8ded1b92f4"} Oct 31 01:22:47.450138 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:47.449942 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:47.562697 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:47.562595 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" podStartSLOduration=1.5625781669999999 podStartE2EDuration="1.562578167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:47.561363898 +0000 UTC m=+3259.763614332" watchObservedRunningTime="2026-10-31 01:22:47.562578167 +0000 UTC m=+3259.764828600" Oct 31 01:22:57.195823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.195765 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd"] Oct 31 01:22:57.200538 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.200516 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.207719 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.207511 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-credential-socket\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.207719 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.207667 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-envoy\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.207887 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.207776 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-token\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.207887 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.207834 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istiod-ca-cert\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.207887 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.207860 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-socket\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.208043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.207938 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-certs\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.208043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.208014 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-podinfo\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.208160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.208069 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gppj2\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-kube-api-access-gppj2\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.208160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.208151 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-data\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.263370 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.263330 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd"] Oct 31 01:22:57.306715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.306684 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-9332a5-openshift-default-dockercfg-5q2kj\"" Oct 31 01:22:57.309496 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309463 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gppj2\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-kube-api-access-gppj2\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.309620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309516 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-data\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.309620 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309568 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-credential-socket\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.309724 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309653 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-envoy\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.309724 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309705 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-token\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.309851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309736 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istiod-ca-cert\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.309851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309762 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-socket\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.309851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309810 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-certs\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.310016 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.309878 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-podinfo\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.310077 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.310007 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-data\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.310077 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.310052 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-credential-socket\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.310311 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.310259 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-socket\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.310441 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.310306 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-certs\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.310666 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.310640 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istiod-ca-cert\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.312406 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.312383 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-podinfo\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.312487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.312450 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-envoy\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.391532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.391497 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-token\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.401161 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.401121 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gppj2\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-kube-api-access-gppj2\") pod \"e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.513911 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.513812 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:57.941084 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.941051 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd"] Oct 31 01:22:57.941264 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:22:57.941243 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d45daef_a5b8_4c2f_9b19_9e74a27288e2.slice/crio-ef93e9a6559b7b2a781c8c1e274ebe6cf44a254607c3c30b3002bc8b3376df67 WatchSource:0}: Error finding container ef93e9a6559b7b2a781c8c1e274ebe6cf44a254607c3c30b3002bc8b3376df67: Status 404 returned error can't find the container with id ef93e9a6559b7b2a781c8c1e274ebe6cf44a254607c3c30b3002bc8b3376df67 Oct 31 01:22:57.943613 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.943576 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:57.943733 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.943655 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:57.943733 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:57.943700 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:58.460321 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:58.460292 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:22:58.532869 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:58.532822 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" event={"ID":"6d45daef-a5b8-4c2f-9b19-9e74a27288e2","Type":"ContainerStarted","Data":"1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699"} Oct 31 01:22:58.533044 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:58.532922 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" event={"ID":"6d45daef-a5b8-4c2f-9b19-9e74a27288e2","Type":"ContainerStarted","Data":"ef93e9a6559b7b2a781c8c1e274ebe6cf44a254607c3c30b3002bc8b3376df67"} Oct 31 01:22:58.691733 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:58.691675 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podStartSLOduration=1.691658244 podStartE2EDuration="1.691658244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:58.689398468 +0000 UTC m=+3270.891648902" watchObservedRunningTime="2026-10-31 01:22:58.691658244 +0000 UTC m=+3270.893908764" Oct 31 01:22:59.513981 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:59.513953 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:59.522121 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:59.522075 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:59.539527 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:59.539495 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:22:59.540271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:22:59.540251 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:23:01.128704 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.128642 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" podUID="98ae480a-cc18-4970-9853-b8ec4191156a" containerName="main" containerID="cri-o://7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f" gracePeriod=45 Oct 31 01:23:01.430938 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.430912 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:23:01.474072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474039 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-model-cache\") pod \"98ae480a-cc18-4970-9853-b8ec4191156a\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " Oct 31 01:23:01.474252 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474086 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ae480a-cc18-4970-9853-b8ec4191156a-kube-api-access-dvh57\" (UniqueName: \"kubernetes.io/projected/98ae480a-cc18-4970-9853-b8ec4191156a-kube-api-access-dvh57\") pod \"98ae480a-cc18-4970-9853-b8ec4191156a\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " Oct 31 01:23:01.474252 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474231 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ae480a-cc18-4970-9853-b8ec4191156a-tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae480a-cc18-4970-9853-b8ec4191156a-tls-certs\") pod \"98ae480a-cc18-4970-9853-b8ec4191156a\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " Oct 31 01:23:01.474392 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474296 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-model-cache" pod "98ae480a-cc18-4970-9853-b8ec4191156a" (UID: "98ae480a-cc18-4970-9853-b8ec4191156a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:01.474392 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474320 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-home\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-home\") pod \"98ae480a-cc18-4970-9853-b8ec4191156a\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " Oct 31 01:23:01.474392 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474357 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-tmp-dir\") pod \"98ae480a-cc18-4970-9853-b8ec4191156a\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " Oct 31 01:23:01.474551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474444 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-dshm\") pod \"98ae480a-cc18-4970-9853-b8ec4191156a\" (UID: \"98ae480a-cc18-4970-9853-b8ec4191156a\") " Oct 31 01:23:01.474679 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474641 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-home" pod "98ae480a-cc18-4970-9853-b8ec4191156a" (UID: "98ae480a-cc18-4970-9853-b8ec4191156a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:01.474772 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.474730 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-tmp-dir" pod "98ae480a-cc18-4970-9853-b8ec4191156a" (UID: "98ae480a-cc18-4970-9853-b8ec4191156a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:01.475048 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.475025 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:01.475048 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.475047 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:01.475238 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.475062 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:01.476633 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.476610 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ae480a-cc18-4970-9853-b8ec4191156a-tls-certs" pod "98ae480a-cc18-4970-9853-b8ec4191156a" (UID: "98ae480a-cc18-4970-9853-b8ec4191156a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:01.476820 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.476770 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-dshm" pod "98ae480a-cc18-4970-9853-b8ec4191156a" (UID: "98ae480a-cc18-4970-9853-b8ec4191156a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:01.477127 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.477109 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ae480a-cc18-4970-9853-b8ec4191156a-kube-api-access-dvh57" pod "98ae480a-cc18-4970-9853-b8ec4191156a" (UID: "98ae480a-cc18-4970-9853-b8ec4191156a"). InnerVolumeSpecName "kube-api-access-dvh57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:01.558573 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.558533 5890 generic.go:356] "Generic (PLEG): container finished" podID="98ae480a-cc18-4970-9853-b8ec4191156a" containerID="7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f" exitCode=0 Oct 31 01:23:01.558771 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.558589 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" Oct 31 01:23:01.558771 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.558607 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" event={"ID":"98ae480a-cc18-4970-9853-b8ec4191156a","Type":"ContainerDied","Data":"7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f"} Oct 31 01:23:01.558771 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.558659 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd" event={"ID":"98ae480a-cc18-4970-9853-b8ec4191156a","Type":"ContainerDied","Data":"99928504034d81c21634d91d96b5f175725e71f9f8781e2f4acef7cdb48002f1"} Oct 31 01:23:01.558771 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.558684 5890 scope.go:122] "RemoveContainer" containerID="7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f" Oct 31 01:23:01.573327 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.573302 5890 scope.go:122] "RemoveContainer" containerID="7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f" Oct 31 01:23:01.573661 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:23:01.573635 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f\": container with ID starting with 7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f not found: ID does not exist" containerID="7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f" Oct 31 01:23:01.573737 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.573668 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f"} err="failed to get container status \"7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f\": rpc error: code = NotFound desc = could not find container \"7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f\": container with ID starting with 7913f681098722a0a469dc0caad5e903734fbc786ea84276f3e9393f3bb3e03f not found: ID does not exist" Oct 31 01:23:01.576497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.576471 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae480a-cc18-4970-9853-b8ec4191156a-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:01.576497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.576496 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae480a-cc18-4970-9853-b8ec4191156a-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:01.576682 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.576506 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvh57\" (UniqueName: \"kubernetes.io/projected/98ae480a-cc18-4970-9853-b8ec4191156a-kube-api-access-dvh57\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:01.717773 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.717737 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd"] Oct 31 01:23:01.724150 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.724116 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9925a/auth-test-model-f9925a-kserve-569cfc9cf7-b8mqd"] Oct 31 01:23:01.935010 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:01.934978 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ae480a-cc18-4970-9853-b8ec4191156a" path="/var/lib/kubelet/pods/98ae480a-cc18-4970-9853-b8ec4191156a/volumes" Oct 31 01:23:02.022172 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.022089 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" podUID="3b669c9b-3cb1-4aa5-accb-be9f8305fd85" containerName="main" containerID="cri-o://c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716" gracePeriod=45 Oct 31 01:23:02.100673 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.100630 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" podUID="3b669c9b-3cb1-4aa5-accb-be9f8305fd85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": dial tcp 10.132.0.165:8000: connect: connection refused" Oct 31 01:23:02.305167 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.305140 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:23:02.390597 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.390406 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tmp-dir\") pod \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " Oct 31 01:23:02.390597 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.390456 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-kube-api-access-h2stt\") pod \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " Oct 31 01:23:02.390597 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.390501 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tls-certs\") pod \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " Oct 31 01:23:02.390597 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.390544 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-model-cache\") pod \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " Oct 31 01:23:02.390597 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.390579 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-home\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-home\") pod \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " Oct 31 01:23:02.391106 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.390721 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-dshm\") pod \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\" (UID: \"3b669c9b-3cb1-4aa5-accb-be9f8305fd85\") " Oct 31 01:23:02.398072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.394907 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tmp-dir" pod "3b669c9b-3cb1-4aa5-accb-be9f8305fd85" (UID: "3b669c9b-3cb1-4aa5-accb-be9f8305fd85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:02.398072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.394943 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-model-cache" pod "3b669c9b-3cb1-4aa5-accb-be9f8305fd85" (UID: "3b669c9b-3cb1-4aa5-accb-be9f8305fd85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:02.398072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.395005 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-home" pod "3b669c9b-3cb1-4aa5-accb-be9f8305fd85" (UID: "3b669c9b-3cb1-4aa5-accb-be9f8305fd85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:02.398072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.395314 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-dshm" pod "3b669c9b-3cb1-4aa5-accb-be9f8305fd85" (UID: "3b669c9b-3cb1-4aa5-accb-be9f8305fd85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:02.400138 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.400098 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tls-certs" pod "3b669c9b-3cb1-4aa5-accb-be9f8305fd85" (UID: "3b669c9b-3cb1-4aa5-accb-be9f8305fd85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:02.400255 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.400171 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-kube-api-access-h2stt" pod "3b669c9b-3cb1-4aa5-accb-be9f8305fd85" (UID: "3b669c9b-3cb1-4aa5-accb-be9f8305fd85"). InnerVolumeSpecName "kube-api-access-h2stt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:02.492661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.492623 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:02.492661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.492664 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:02.492883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.492679 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2stt\" (UniqueName: \"kubernetes.io/projected/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-kube-api-access-h2stt\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:02.492883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.492693 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:02.492883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.492707 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:02.492883 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.492722 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b669c9b-3cb1-4aa5-accb-be9f8305fd85-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:23:02.566555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.566442 5890 generic.go:356] "Generic (PLEG): container finished" podID="3b669c9b-3cb1-4aa5-accb-be9f8305fd85" containerID="c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716" exitCode=0 Oct 31 01:23:02.566555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.566496 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" Oct 31 01:23:02.566555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.566531 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" event={"ID":"3b669c9b-3cb1-4aa5-accb-be9f8305fd85","Type":"ContainerDied","Data":"c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716"} Oct 31 01:23:02.566874 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.566585 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt" event={"ID":"3b669c9b-3cb1-4aa5-accb-be9f8305fd85","Type":"ContainerDied","Data":"60b66e7f85bbb5d7f2152f4aa8dca83215b2488facac7c512ad81d168cdcd4ff"} Oct 31 01:23:02.566874 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.566608 5890 scope.go:122] "RemoveContainer" containerID="c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716" Oct 31 01:23:02.585554 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.585527 5890 scope.go:122] "RemoveContainer" containerID="c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716" Oct 31 01:23:02.586002 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:23:02.585972 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716\": container with ID starting with c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716 not found: ID does not exist" containerID="c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716" Oct 31 01:23:02.586106 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.586011 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716"} err="failed to get container status \"c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716\": rpc error: code = NotFound desc = could not find container \"c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716\": container with ID starting with c89b646beff0edd67a31fcc2560cde84551a9005d785c85deaa49f1cadc4f716 not found: ID does not exist" Oct 31 01:23:02.744609 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.744569 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt"] Oct 31 01:23:02.752054 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:02.752022 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7f8596/auth-test-model-7f8596-kserve-5ff88d84bf-qsjtt"] Oct 31 01:23:03.933880 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:03.933848 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b669c9b-3cb1-4aa5-accb-be9f8305fd85" path="/var/lib/kubelet/pods/3b669c9b-3cb1-4aa5-accb-be9f8305fd85/volumes" Oct 31 01:23:07.018623 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:23:07.018581 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:23:12.150740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.150696 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd"] Oct 31 01:23:12.153146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.151462 5890 cpu_manager.go:414] "containerMap: removing container" podUID="98ae480a-cc18-4970-9853-b8ec4191156a" containerName="main" Oct 31 01:23:12.153146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.151481 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ae480a-cc18-4970-9853-b8ec4191156a" containerName="main" Oct 31 01:23:12.153146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.151531 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3b669c9b-3cb1-4aa5-accb-be9f8305fd85" containerName="main" Oct 31 01:23:12.153146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.151536 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b669c9b-3cb1-4aa5-accb-be9f8305fd85" containerName="main" Oct 31 01:23:12.153146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.151621 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b669c9b-3cb1-4aa5-accb-be9f8305fd85" containerName="main" Oct 31 01:23:12.153146 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.151631 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ae480a-cc18-4970-9853-b8ec4191156a" containerName="main" Oct 31 01:23:12.154863 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.154842 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.255505 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.255471 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3aca3f-serving-cert\"" Oct 31 01:23:12.260464 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.260432 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd"] Oct 31 01:23:12.317392 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.317356 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.317587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.317512 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.317642 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.317592 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.317679 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.317663 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x2dj\" (UniqueName: \"kubernetes.io/projected/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-kube-api-access-5x2dj\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.418650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.418542 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.418854 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.418651 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2dj\" (UniqueName: \"kubernetes.io/projected/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-kube-api-access-5x2dj\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.418854 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.418771 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.419016 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.418866 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.419016 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:23:12.418962 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-serving-cert: secret "maas-api-e2e-extmod-a-3aca3f-serving-cert" not found Oct 31 01:23:12.419016 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:23:12.419017 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls podName:746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c nodeName:}" failed. No retries permitted until 2026-10-31 01:23:12.919002601 +0000 UTC m=+3285.121253011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls") pod "maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" (UID: "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c") : secret "maas-api-e2e-extmod-a-3aca3f-serving-cert" not found Oct 31 01:23:12.419323 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.419300 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.421377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.421351 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.483672 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.482919 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2dj\" (UniqueName: \"kubernetes.io/projected/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-kube-api-access-5x2dj\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.925535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.925500 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:12.927779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:12.927755 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:13.069939 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:13.069903 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:13.446285 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:13.446246 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd"] Oct 31 01:23:13.446709 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:23:13.446678 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746ebe0d_c46c_4aa6_8f4a_c4c7ad26b24c.slice/crio-411007f90d982db87267c3f9820df530d531543c8f10bda4a880085556044436 WatchSource:0}: Error finding container 411007f90d982db87267c3f9820df530d531543c8f10bda4a880085556044436: Status 404 returned error can't find the container with id 411007f90d982db87267c3f9820df530d531543c8f10bda4a880085556044436 Oct 31 01:23:13.652456 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:13.652405 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" event={"ID":"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c","Type":"ContainerStarted","Data":"411007f90d982db87267c3f9820df530d531543c8f10bda4a880085556044436"} Oct 31 01:23:14.661262 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:14.661222 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" event={"ID":"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c","Type":"ContainerStarted","Data":"bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1"} Oct 31 01:23:14.661675 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:14.661301 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:14.777246 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:14.777193 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" podStartSLOduration=1.805773144 podStartE2EDuration="2.77717942s" totalImagesPullingTime="971.406276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:14.776090101 +0000 UTC m=+3286.978340535" watchObservedRunningTime="2026-10-31 01:23:14.77717942 +0000 UTC m=+3286.979429854" Oct 31 01:23:20.676185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:20.676154 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:23:28.981269 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:28.981233 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:23:29.102330 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:29.102302 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:23:37.018874 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:23:37.018829 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:23:44.881705 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:44.881671 5890 generic.go:356] "Generic (PLEG): container finished" podID="a5016469-de22-402b-b73a-471dab674b25" containerID="be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f" exitCode=0 Oct 31 01:23:44.882189 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:44.881748 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" event={"ID":"a5016469-de22-402b-b73a-471dab674b25","Type":"ContainerDied","Data":"be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f"} Oct 31 01:23:44.882189 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:44.882178 5890 scope.go:122] "RemoveContainer" containerID="be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f" Oct 31 01:23:44.883877 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:44.883842 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:23:44.883984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:44.883916 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:23:44.883984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:44.883948 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:23:45.893331 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:45.893288 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" event={"ID":"a5016469-de22-402b-b73a-471dab674b25","Type":"ContainerStarted","Data":"b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0"} Oct 31 01:23:46.277166 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:46.277125 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:23:46.277398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:46.277251 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 01:23:46.277398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:46.277296 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 01:23:47.279336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:47.279297 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 01:23:47.279754 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:47.279360 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 01:23:48.276810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:48.276653 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 01:23:48.276810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:48.276715 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 01:23:49.274191 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:49.274144 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 01:23:49.274722 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:49.274212 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 01:23:50.275257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:50.275214 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 01:23:50.275818 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:50.275313 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 01:23:51.274258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:51.274205 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 01:23:51.274523 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:51.274290 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 01:23:52.279421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:52.279384 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:23:52.279922 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:52.279806 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:23:52.280646 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:23:52.280623 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:24:00.394655 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.394621 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh"] Oct 31 01:24:00.397776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.397755 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.454240 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.454202 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.454565 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.454535 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3629424-1787-406c-9bee-01f5fe05ff24-openshift-service-ca\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.454765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.454726 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzgh\" (UniqueName: \"kubernetes.io/projected/e3629424-1787-406c-9bee-01f5fe05ff24-kube-api-access-9kzgh\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.454894 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.454769 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-metrics-certs\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.499214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.499184 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-6b5091-serving-cert\"" Oct 31 01:24:00.502351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.502325 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh"] Oct 31 01:24:00.555651 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.555619 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.555858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.555707 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3629424-1787-406c-9bee-01f5fe05ff24-openshift-service-ca\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.555858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.555754 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzgh\" (UniqueName: \"kubernetes.io/projected/e3629424-1787-406c-9bee-01f5fe05ff24-kube-api-access-9kzgh\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.555858 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.555779 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-metrics-certs\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.556049 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:24:00.555797 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-serving-cert: secret "maas-api-e2e-autores-6b5091-serving-cert" not found Oct 31 01:24:00.556049 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:24:00.555985 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls podName:e3629424-1787-406c-9bee-01f5fe05ff24 nodeName:}" failed. No retries permitted until 2026-10-31 01:24:01.055957726 +0000 UTC m=+3333.258208153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls") pod "maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" (UID: "e3629424-1787-406c-9bee-01f5fe05ff24") : secret "maas-api-e2e-autores-6b5091-serving-cert" not found Oct 31 01:24:00.556502 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.556476 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3629424-1787-406c-9bee-01f5fe05ff24-openshift-service-ca\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.558451 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.558430 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-metrics-certs\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:00.617342 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:00.617318 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzgh\" (UniqueName: \"kubernetes.io/projected/e3629424-1787-406c-9bee-01f5fe05ff24-kube-api-access-9kzgh\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:01.062336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:01.062304 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:01.064776 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:01.064745 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls\") pod \"maas-api-e2e-autores-6b5091-7b79777f7c-7njfh\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:01.311316 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:01.311279 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:01.695486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:01.695451 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh"] Oct 31 01:24:01.696444 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:24:01.696410 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3629424_1787_406c_9bee_01f5fe05ff24.slice/crio-4c3c8feda50055cd371f170c6d7d8f416e885ebf8333c1769c9e2ba3a55680f3 WatchSource:0}: Error finding container 4c3c8feda50055cd371f170c6d7d8f416e885ebf8333c1769c9e2ba3a55680f3: Status 404 returned error can't find the container with id 4c3c8feda50055cd371f170c6d7d8f416e885ebf8333c1769c9e2ba3a55680f3 Oct 31 01:24:02.008071 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:02.007989 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" event={"ID":"e3629424-1787-406c-9bee-01f5fe05ff24","Type":"ContainerStarted","Data":"4c3c8feda50055cd371f170c6d7d8f416e885ebf8333c1769c9e2ba3a55680f3"} Oct 31 01:24:03.016507 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:03.016460 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" event={"ID":"e3629424-1787-406c-9bee-01f5fe05ff24","Type":"ContainerStarted","Data":"40997376e116f13003efa2d23d8ac1eda5acba38f3b49ce4f92b654df7925dd3"} Oct 31 01:24:03.016950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:03.016535 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:03.131358 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:03.131308 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" podStartSLOduration=2.32237949 podStartE2EDuration="3.131295034s" totalImagesPullingTime="808.915544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:03.130386275 +0000 UTC m=+3335.332636709" watchObservedRunningTime="2026-10-31 01:24:03.131295034 +0000 UTC m=+3335.333545466" Oct 31 01:24:07.018263 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:24:07.018221 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:24:09.028728 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:09.028649 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:24:19.709996 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.709957 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n"] Oct 31 01:24:19.714919 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.714896 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.774909 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.774879 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-08fcc4-serving-cert\"" Oct 31 01:24:19.812759 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.812723 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.812954 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.812767 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-metrics-certs\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.812954 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.812927 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqssf\" (UniqueName: \"kubernetes.io/projected/fc8301ab-23c3-4998-969c-153ff79dd3e4-kube-api-access-kqssf\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.813043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.813027 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8301ab-23c3-4998-969c-153ff79dd3e4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.824867 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.824843 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n"] Oct 31 01:24:19.914010 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.913971 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.914198 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.914016 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-metrics-certs\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.914198 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.914111 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqssf\" (UniqueName: \"kubernetes.io/projected/fc8301ab-23c3-4998-969c-153ff79dd3e4-kube-api-access-kqssf\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.914198 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:24:19.914139 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-serving-cert: secret "maas-api-e2e-extmod-a-08fcc4-serving-cert" not found Oct 31 01:24:19.914390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.914199 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8301ab-23c3-4998-969c-153ff79dd3e4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.914390 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:24:19.914223 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls podName:fc8301ab-23c3-4998-969c-153ff79dd3e4 nodeName:}" failed. No retries permitted until 2026-10-31 01:24:20.414201834 +0000 UTC m=+3352.616452246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls") pod "maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" (UID: "fc8301ab-23c3-4998-969c-153ff79dd3e4") : secret "maas-api-e2e-extmod-a-08fcc4-serving-cert" not found Oct 31 01:24:19.914986 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.914962 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8301ab-23c3-4998-969c-153ff79dd3e4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.916591 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.916568 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-metrics-certs\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:19.994274 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:19.994191 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqssf\" (UniqueName: \"kubernetes.io/projected/fc8301ab-23c3-4998-969c-153ff79dd3e4-kube-api-access-kqssf\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:20.420691 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:20.420602 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:20.423150 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:20.423122 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:20.628310 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:20.628273 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:21.055836 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:21.043840 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n"] Oct 31 01:24:21.056851 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:24:21.056813 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8301ab_23c3_4998_969c_153ff79dd3e4.slice/crio-514853a2cffa05e3fe5beeb27af308d47c6727530d88126c068ff343f608d068 WatchSource:0}: Error finding container 514853a2cffa05e3fe5beeb27af308d47c6727530d88126c068ff343f608d068: Status 404 returned error can't find the container with id 514853a2cffa05e3fe5beeb27af308d47c6727530d88126c068ff343f608d068 Oct 31 01:24:21.216504 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:21.216468 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" event={"ID":"fc8301ab-23c3-4998-969c-153ff79dd3e4","Type":"ContainerStarted","Data":"514853a2cffa05e3fe5beeb27af308d47c6727530d88126c068ff343f608d068"} Oct 31 01:24:23.235625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:23.235591 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" event={"ID":"fc8301ab-23c3-4998-969c-153ff79dd3e4","Type":"ContainerStarted","Data":"638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22"} Oct 31 01:24:23.236033 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:23.235759 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:23.351114 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:23.351038 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" podStartSLOduration=2.990794085 podStartE2EDuration="4.351018641s" totalImagesPullingTime="1.360224556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:23.349112791 +0000 UTC m=+3355.551363222" watchObservedRunningTime="2026-10-31 01:24:23.351018641 +0000 UTC m=+3355.553269076" Oct 31 01:24:25.258324 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:25.258234 5890 generic.go:356] "Generic (PLEG): container finished" podID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerID="446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b" exitCode=0 Oct 31 01:24:25.258324 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:25.258261 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" event={"ID":"31a53953-ecea-4303-8be1-99b6f93bc3e9","Type":"ContainerDied","Data":"446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b"} Oct 31 01:24:25.258823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:25.258710 5890 scope.go:122] "RemoveContainer" containerID="446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b" Oct 31 01:24:25.261180 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:25.261143 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:24:25.261302 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:25.261240 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:24:25.261302 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:25.261287 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:24:26.286584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:26.286542 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" event={"ID":"31a53953-ecea-4303-8be1-99b6f93bc3e9","Type":"ContainerStarted","Data":"5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2"} Oct 31 01:24:26.390823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:26.376729 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:24:26.390823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:26.376837 5890 patch_prober.go:29] interesting pod/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 31 01:24:26.390823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:26.376926 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 31 01:24:27.377322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:27.377279 5890 patch_prober.go:29] interesting pod/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 31 01:24:27.377760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:27.377362 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 31 01:24:28.377213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:28.377175 5890 patch_prober.go:29] interesting pod/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 31 01:24:28.377400 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:28.377238 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 31 01:24:29.272304 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:29.272272 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:24:29.377403 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:29.377363 5890 patch_prober.go:29] interesting pod/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 31 01:24:29.377857 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:29.377434 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 31 01:24:30.377293 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:30.377252 5890 patch_prober.go:29] interesting pod/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 31 01:24:30.377487 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:30.377313 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 31 01:24:31.376895 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:31.376858 5890 patch_prober.go:29] interesting pod/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 31 01:24:31.377076 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:31.376915 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 31 01:24:32.382920 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:32.382889 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:24:32.383442 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:32.383255 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:24:32.384145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:24:32.384121 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:24:37.019207 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:24:37.019160 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:25:00.581059 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:00.581000 5890 patch_prober.go:29] interesting pod/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:25:00.581616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:00.581103 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:25:02.549757 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:02.549717 5890 generic.go:356] "Generic (PLEG): container finished" podID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerID="1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699" exitCode=0 Oct 31 01:25:02.550255 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:02.549805 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" event={"ID":"6d45daef-a5b8-4c2f-9b19-9e74a27288e2","Type":"ContainerDied","Data":"1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699"} Oct 31 01:25:02.550343 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:02.550326 5890 scope.go:122] "RemoveContainer" containerID="1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699" Oct 31 01:25:02.552372 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:02.552336 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:02.552477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:02.552429 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:02.552523 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:02.552476 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:03.580209 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:03.580169 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" event={"ID":"6d45daef-a5b8-4c2f-9b19-9e74a27288e2","Type":"ContainerStarted","Data":"5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b"} Oct 31 01:25:04.514239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:04.514201 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:25:04.514528 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:04.514504 5890 patch_prober.go:29] interesting pod/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 01:25:04.514595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:04.514553 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 01:25:05.515137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:05.515088 5890 patch_prober.go:29] interesting pod/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 01:25:05.515522 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:05.515149 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 01:25:06.514384 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:06.514349 5890 patch_prober.go:29] interesting pod/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 01:25:06.514567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:06.514400 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 01:25:07.020360 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:25:07.020307 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:25:07.515098 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:07.515063 5890 patch_prober.go:29] interesting pod/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 01:25:07.515281 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:07.515122 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 01:25:08.515004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:08.514969 5890 patch_prober.go:29] interesting pod/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 01:25:08.515406 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:08.515023 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 01:25:09.519315 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:09.519282 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:25:09.519823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:09.519800 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:25:09.520439 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:09.520419 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:25:37.018693 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:25:37.018646 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:25:44.465963 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:44.465924 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k"] Oct 31 01:25:49.466512 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.466458 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" podUID="807f2910-69bb-4094-9027-5bf693b64fb3" containerName="payload-processing" containerID="cri-o://8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a" gracePeriod=25 Oct 31 01:25:49.728315 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.728253 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:25:49.859526 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.859490 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807f2910-69bb-4094-9027-5bf693b64fb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/807f2910-69bb-4094-9027-5bf693b64fb3-plugins-config-volume\") pod \"807f2910-69bb-4094-9027-5bf693b64fb3\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " Oct 31 01:25:49.859866 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.859689 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807f2910-69bb-4094-9027-5bf693b64fb3-kube-api-access-xssrl\" (UniqueName: \"kubernetes.io/projected/807f2910-69bb-4094-9027-5bf693b64fb3-kube-api-access-xssrl\") pod \"807f2910-69bb-4094-9027-5bf693b64fb3\" (UID: \"807f2910-69bb-4094-9027-5bf693b64fb3\") " Oct 31 01:25:49.859991 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.859963 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807f2910-69bb-4094-9027-5bf693b64fb3-plugins-config-volume" pod "807f2910-69bb-4094-9027-5bf693b64fb3" (UID: "807f2910-69bb-4094-9027-5bf693b64fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:49.860225 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.860202 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/807f2910-69bb-4094-9027-5bf693b64fb3-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:49.862076 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.862048 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807f2910-69bb-4094-9027-5bf693b64fb3-kube-api-access-xssrl" pod "807f2910-69bb-4094-9027-5bf693b64fb3" (UID: "807f2910-69bb-4094-9027-5bf693b64fb3"). InnerVolumeSpecName "kube-api-access-xssrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:49.940995 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.940956 5890 generic.go:356] "Generic (PLEG): container finished" podID="807f2910-69bb-4094-9027-5bf693b64fb3" containerID="8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a" exitCode=0 Oct 31 01:25:49.941179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.941009 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" Oct 31 01:25:49.941179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.941013 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" event={"ID":"807f2910-69bb-4094-9027-5bf693b64fb3","Type":"ContainerDied","Data":"8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a"} Oct 31 01:25:49.941314 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.941172 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k" event={"ID":"807f2910-69bb-4094-9027-5bf693b64fb3","Type":"ContainerDied","Data":"c5407dd10eebe110af6f7d662b81ccc053cbdad2e2a16db5a47eac8ded1b92f4"} Oct 31 01:25:49.941314 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.941214 5890 scope.go:122] "RemoveContainer" containerID="8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a" Oct 31 01:25:49.953411 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.953391 5890 scope.go:122] "RemoveContainer" containerID="8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a" Oct 31 01:25:49.953683 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:25:49.953666 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a\": container with ID starting with 8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a not found: ID does not exist" containerID="8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a" Oct 31 01:25:49.953740 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.953690 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a"} err="failed to get container status \"8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a\": rpc error: code = NotFound desc = could not find container \"8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a\": container with ID starting with 8d031855ab5327e1daec67b8456c83c498cda816ec40936123303807510fac9a not found: ID does not exist" Oct 31 01:25:49.961428 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:49.961395 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xssrl\" (UniqueName: \"kubernetes.io/projected/807f2910-69bb-4094-9027-5bf693b64fb3-kube-api-access-xssrl\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:50.123602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:50.123220 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k"] Oct 31 01:25:50.126509 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:50.126481 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6b5091-557fb7f58c-l8l7k"] Oct 31 01:25:51.933605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:25:51.933575 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807f2910-69bb-4094-9027-5bf693b64fb3" path="/var/lib/kubelet/pods/807f2910-69bb-4094-9027-5bf693b64fb3/volumes" Oct 31 01:26:07.018198 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:26:07.018160 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:26:12.664332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:12.664291 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z"] Oct 31 01:26:17.664905 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:17.664838 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" podUID="276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" containerName="payload-pre-processing" containerID="cri-o://09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589" gracePeriod=25 Oct 31 01:26:17.952611 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:17.952576 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:26:18.119639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.119607 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-plugins-config-volume\") pod \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " Oct 31 01:26:18.119830 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.119820 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-kube-api-access-cksjs\" (UniqueName: \"kubernetes.io/projected/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-kube-api-access-cksjs\") pod \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\" (UID: \"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7\") " Oct 31 01:26:18.120002 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.119978 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-plugins-config-volume" pod "276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" (UID: "276eb75d-7f2b-4451-bd5f-a7fc8ded55a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:18.120295 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.120277 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:18.121904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.121878 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-kube-api-access-cksjs" pod "276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" (UID: "276eb75d-7f2b-4451-bd5f-a7fc8ded55a7"). InnerVolumeSpecName "kube-api-access-cksjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:18.168401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.168368 5890 generic.go:356] "Generic (PLEG): container finished" podID="276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" containerID="09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589" exitCode=0 Oct 31 01:26:18.168537 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.168418 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" Oct 31 01:26:18.168537 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.168450 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" event={"ID":"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7","Type":"ContainerDied","Data":"09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589"} Oct 31 01:26:18.168537 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.168513 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z" event={"ID":"276eb75d-7f2b-4451-bd5f-a7fc8ded55a7","Type":"ContainerDied","Data":"d993dac955d0061465ff6a9b6441ca2b3708838c39febbcb9dbd81216371395c"} Oct 31 01:26:18.168704 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.168544 5890 scope.go:122] "RemoveContainer" containerID="09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589" Oct 31 01:26:18.179915 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.179844 5890 scope.go:122] "RemoveContainer" containerID="09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589" Oct 31 01:26:18.180152 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:26:18.180133 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589\": container with ID starting with 09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589 not found: ID does not exist" containerID="09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589" Oct 31 01:26:18.180196 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.180160 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589"} err="failed to get container status \"09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589\": rpc error: code = NotFound desc = could not find container \"09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589\": container with ID starting with 09846feb111395f1195d85f282f28d1aa091361047464d682ea77f7e1d2e7589 not found: ID does not exist" Oct 31 01:26:18.221112 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.221088 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksjs\" (UniqueName: \"kubernetes.io/projected/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7-kube-api-access-cksjs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:18.325701 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.325666 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z"] Oct 31 01:26:18.331569 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:18.331539 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb68dd-6d6bfd955f-jrt8z"] Oct 31 01:26:19.934698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:19.934663 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" path="/var/lib/kubelet/pods/276eb75d-7f2b-4451-bd5f-a7fc8ded55a7/volumes" Oct 31 01:26:28.809943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:28.809908 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd"] Oct 31 01:26:28.810499 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:28.810131 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" podUID="746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" containerName="maas-api" containerID="cri-o://bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1" gracePeriod=30 Oct 31 01:26:29.114476 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.114451 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:26:29.159628 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.159597 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-openshift-service-ca\") pod \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " Oct 31 01:26:29.159830 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.159814 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-kube-api-access-5x2dj\" (UniqueName: \"kubernetes.io/projected/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-kube-api-access-5x2dj\") pod \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " Oct 31 01:26:29.159918 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.159891 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-metrics-certs\") pod \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " Oct 31 01:26:29.159983 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.159940 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls\") pod \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\" (UID: \"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c\") " Oct 31 01:26:29.160042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.159985 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-openshift-service-ca" pod "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" (UID: "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:29.160371 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.160353 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:29.161893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.161866 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-kube-api-access-5x2dj" pod "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" (UID: "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c"). InnerVolumeSpecName "kube-api-access-5x2dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:29.162031 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.162010 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls" pod "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" (UID: "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:29.162250 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.162235 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-metrics-certs" pod "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" (UID: "746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:29.253293 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.253260 5890 generic.go:356] "Generic (PLEG): container finished" podID="746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" containerID="bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1" exitCode=0 Oct 31 01:26:29.253486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.253327 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" Oct 31 01:26:29.253486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.253342 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" event={"ID":"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c","Type":"ContainerDied","Data":"bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1"} Oct 31 01:26:29.253486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.253398 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd" event={"ID":"746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c","Type":"ContainerDied","Data":"411007f90d982db87267c3f9820df530d531543c8f10bda4a880085556044436"} Oct 31 01:26:29.253486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.253420 5890 scope.go:122] "RemoveContainer" containerID="bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1" Oct 31 01:26:29.261676 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.261644 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x2dj\" (UniqueName: \"kubernetes.io/projected/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-kube-api-access-5x2dj\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:29.261827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.261679 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:29.261827 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.261696 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:29.264460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.264443 5890 scope.go:122] "RemoveContainer" containerID="bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1" Oct 31 01:26:29.264918 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:26:29.264751 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1\": container with ID starting with bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1 not found: ID does not exist" containerID="bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1" Oct 31 01:26:29.264918 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.264800 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1"} err="failed to get container status \"bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1\": rpc error: code = NotFound desc = could not find container \"bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1\": container with ID starting with bc998733234cd688acf134c8f45bc826cc57289694a732c2418e54b5ce3f51c1 not found: ID does not exist" Oct 31 01:26:29.456120 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.456088 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd"] Oct 31 01:26:29.464840 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.464775 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3aca3f-798ddd7d5-fmfrd"] Oct 31 01:26:29.935736 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:29.935695 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" path="/var/lib/kubelet/pods/746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c/volumes" Oct 31 01:26:37.019537 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:26:37.019496 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:26:38.048282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:38.048242 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm"] Oct 31 01:26:38.048741 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:38.048518 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" containerID="cri-o://b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0" gracePeriod=30 Oct 31 01:26:43.332512 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.332452 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:26:43.354423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.354381 5890 generic.go:356] "Generic (PLEG): container finished" podID="a5016469-de22-402b-b73a-471dab674b25" containerID="b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0" exitCode=0 Oct 31 01:26:43.354629 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.354610 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" event={"ID":"a5016469-de22-402b-b73a-471dab674b25","Type":"ContainerDied","Data":"b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0"} Oct 31 01:26:43.354720 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.354697 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" event={"ID":"a5016469-de22-402b-b73a-471dab674b25","Type":"ContainerDied","Data":"bbdf08b2aa562b55b4c4e4028a1c78db6243f2a2a741af890758542fdf319159"} Oct 31 01:26:43.354779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.354740 5890 scope.go:122] "RemoveContainer" containerID="b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0" Oct 31 01:26:43.355173 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.354965 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm" Oct 31 01:26:43.367056 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.367028 5890 scope.go:122] "RemoveContainer" containerID="be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f" Oct 31 01:26:43.376664 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.376640 5890 scope.go:122] "RemoveContainer" containerID="b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0" Oct 31 01:26:43.377002 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:26:43.376952 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0\": container with ID starting with b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0 not found: ID does not exist" containerID="b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0" Oct 31 01:26:43.377084 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.377015 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0"} err="failed to get container status \"b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0\": rpc error: code = NotFound desc = could not find container \"b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0\": container with ID starting with b48a9a67f0edfe6dc6be76deec8bec1d8197a5e9160d62e85b68d495ed0ecda0 not found: ID does not exist" Oct 31 01:26:43.377084 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.377040 5890 scope.go:122] "RemoveContainer" containerID="be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f" Oct 31 01:26:43.377319 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:26:43.377274 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f\": container with ID starting with be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f not found: ID does not exist" containerID="be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f" Oct 31 01:26:43.377319 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.377304 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f"} err="failed to get container status \"be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f\": rpc error: code = NotFound desc = could not find container \"be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f\": container with ID starting with be65205a8a8a3f34e30a06619f6578fb1bde1e9276ac766b01419a1401375c6f not found: ID does not exist" Oct 31 01:26:43.454156 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454121 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-socket\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454309 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454223 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-envoy\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454359 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454326 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-kube-api-access-jts9k\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-kube-api-access-jts9k\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454409 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454372 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5016469-de22-402b-b73a-471dab674b25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5016469-de22-402b-b73a-471dab674b25-istiod-ca-cert\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454468 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454416 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-data\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454468 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454430 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-socket" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:43.454587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454466 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-istio-token\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-istio-token\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454509 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-credential-socket\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454560 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-certs\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454748 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454618 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5016469-de22-402b-b73a-471dab674b25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5016469-de22-402b-b73a-471dab674b25-istio-podinfo\") pod \"a5016469-de22-402b-b73a-471dab674b25\" (UID: \"a5016469-de22-402b-b73a-471dab674b25\") " Oct 31 01:26:43.454840 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454823 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5016469-de22-402b-b73a-471dab674b25-istiod-ca-cert" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:43.454988 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.454959 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-data" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:43.455262 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.455236 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-credential-socket" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:43.455382 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.455319 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5016469-de22-402b-b73a-471dab674b25-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.455382 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.455342 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.455382 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.455359 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.455382 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.455371 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-certs" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:43.457111 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.457063 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5016469-de22-402b-b73a-471dab674b25-istio-podinfo" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:26:43.457221 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.457158 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-envoy" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:43.457221 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.457184 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-kube-api-access-jts9k" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "kube-api-access-jts9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:43.457529 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.457508 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-istio-token" pod "a5016469-de22-402b-b73a-471dab674b25" (UID: "a5016469-de22-402b-b73a-471dab674b25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:43.556871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.556838 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.556871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.556868 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jts9k\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-kube-api-access-jts9k\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.557075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.556881 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5016469-de22-402b-b73a-471dab674b25-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.557075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.556890 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.557075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.556900 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5016469-de22-402b-b73a-471dab674b25-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.557075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.556909 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5016469-de22-402b-b73a-471dab674b25-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.773757 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.773719 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm"] Oct 31 01:26:43.777519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.777488 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3aca3f-openshift-default-5b88c4d569-chckm"] Oct 31 01:26:43.933485 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:26:43.933456 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5016469-de22-402b-b73a-471dab674b25" path="/var/lib/kubelet/pods/a5016469-de22-402b-b73a-471dab674b25/volumes" Oct 31 01:27:07.019495 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:07.019459 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:27:10.382996 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.382778 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh"] Oct 31 01:27:10.383892 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.383845 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" podUID="e3629424-1787-406c-9bee-01f5fe05ff24" containerName="maas-api" containerID="cri-o://40997376e116f13003efa2d23d8ac1eda5acba38f3b49ce4f92b654df7925dd3" gracePeriod=30 Oct 31 01:27:10.540602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.540567 5890 generic.go:356] "Generic (PLEG): container finished" podID="e3629424-1787-406c-9bee-01f5fe05ff24" containerID="40997376e116f13003efa2d23d8ac1eda5acba38f3b49ce4f92b654df7925dd3" exitCode=0 Oct 31 01:27:10.540778 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.540648 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" event={"ID":"e3629424-1787-406c-9bee-01f5fe05ff24","Type":"ContainerDied","Data":"40997376e116f13003efa2d23d8ac1eda5acba38f3b49ce4f92b654df7925dd3"} Oct 31 01:27:10.664617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.664590 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:27:10.810891 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.810845 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3629424-1787-406c-9bee-01f5fe05ff24-kube-api-access-9kzgh\" (UniqueName: \"kubernetes.io/projected/e3629424-1787-406c-9bee-01f5fe05ff24-kube-api-access-9kzgh\") pod \"e3629424-1787-406c-9bee-01f5fe05ff24\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " Oct 31 01:27:10.811094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.810953 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3629424-1787-406c-9bee-01f5fe05ff24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3629424-1787-406c-9bee-01f5fe05ff24-openshift-service-ca\") pod \"e3629424-1787-406c-9bee-01f5fe05ff24\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " Oct 31 01:27:10.811094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.811041 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls\") pod \"e3629424-1787-406c-9bee-01f5fe05ff24\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " Oct 31 01:27:10.811215 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.811116 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-metrics-certs\") pod \"e3629424-1787-406c-9bee-01f5fe05ff24\" (UID: \"e3629424-1787-406c-9bee-01f5fe05ff24\") " Oct 31 01:27:10.812062 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.812029 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3629424-1787-406c-9bee-01f5fe05ff24-openshift-service-ca" pod "e3629424-1787-406c-9bee-01f5fe05ff24" (UID: "e3629424-1787-406c-9bee-01f5fe05ff24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:10.813720 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.813685 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3629424-1787-406c-9bee-01f5fe05ff24-kube-api-access-9kzgh" pod "e3629424-1787-406c-9bee-01f5fe05ff24" (UID: "e3629424-1787-406c-9bee-01f5fe05ff24"). InnerVolumeSpecName "kube-api-access-9kzgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:10.814247 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.814216 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-metrics-certs" pod "e3629424-1787-406c-9bee-01f5fe05ff24" (UID: "e3629424-1787-406c-9bee-01f5fe05ff24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:10.816924 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.816896 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls" pod "e3629424-1787-406c-9bee-01f5fe05ff24" (UID: "e3629424-1787-406c-9bee-01f5fe05ff24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:10.913450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.913367 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:10.913450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.913420 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3629424-1787-406c-9bee-01f5fe05ff24-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:10.913450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.913437 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzgh\" (UniqueName: \"kubernetes.io/projected/e3629424-1787-406c-9bee-01f5fe05ff24-kube-api-access-9kzgh\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:10.913673 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:10.913453 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3629424-1787-406c-9bee-01f5fe05ff24-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:11.549615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:11.549581 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" Oct 31 01:27:11.550160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:11.549586 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh" event={"ID":"e3629424-1787-406c-9bee-01f5fe05ff24","Type":"ContainerDied","Data":"4c3c8feda50055cd371f170c6d7d8f416e885ebf8333c1769c9e2ba3a55680f3"} Oct 31 01:27:11.550160 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:11.549720 5890 scope.go:122] "RemoveContainer" containerID="40997376e116f13003efa2d23d8ac1eda5acba38f3b49ce4f92b654df7925dd3" Oct 31 01:27:11.707911 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:11.707872 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh"] Oct 31 01:27:11.755316 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:11.755282 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b5091-7b79777f7c-7njfh"] Oct 31 01:27:11.934739 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:11.934706 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3629424-1787-406c-9bee-01f5fe05ff24" path="/var/lib/kubelet/pods/e3629424-1787-406c-9bee-01f5fe05ff24/volumes" Oct 31 01:27:20.199757 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:20.199719 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr"] Oct 31 01:27:20.200208 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:20.199983 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" containerID="cri-o://5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2" gracePeriod=30 Oct 31 01:27:21.700679 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:21.700637 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p"] Oct 31 01:27:21.701405 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:21.700901 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" podUID="33658ba2-9a0f-4edd-9559-7023b7841b73" containerName="maas-api" containerID="cri-o://d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5" gracePeriod=30 Oct 31 01:27:21.991099 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:21.991069 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:27:22.057632 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.057592 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-metrics-certs\") pod \"33658ba2-9a0f-4edd-9559-7023b7841b73\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " Oct 31 01:27:22.057807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.057753 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33658ba2-9a0f-4edd-9559-7023b7841b73-kube-api-access-pzmm5\" (UniqueName: \"kubernetes.io/projected/33658ba2-9a0f-4edd-9559-7023b7841b73-kube-api-access-pzmm5\") pod \"33658ba2-9a0f-4edd-9559-7023b7841b73\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " Oct 31 01:27:22.057905 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.057891 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls\") pod \"33658ba2-9a0f-4edd-9559-7023b7841b73\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " Oct 31 01:27:22.057944 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.057931 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33658ba2-9a0f-4edd-9559-7023b7841b73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33658ba2-9a0f-4edd-9559-7023b7841b73-openshift-service-ca\") pod \"33658ba2-9a0f-4edd-9559-7023b7841b73\" (UID: \"33658ba2-9a0f-4edd-9559-7023b7841b73\") " Oct 31 01:27:22.058365 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.058333 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33658ba2-9a0f-4edd-9559-7023b7841b73-openshift-service-ca" pod "33658ba2-9a0f-4edd-9559-7023b7841b73" (UID: "33658ba2-9a0f-4edd-9559-7023b7841b73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:22.058525 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.058506 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33658ba2-9a0f-4edd-9559-7023b7841b73-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:22.059982 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.059958 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-metrics-certs" pod "33658ba2-9a0f-4edd-9559-7023b7841b73" (UID: "33658ba2-9a0f-4edd-9559-7023b7841b73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:22.059982 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.059977 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33658ba2-9a0f-4edd-9559-7023b7841b73-kube-api-access-pzmm5" pod "33658ba2-9a0f-4edd-9559-7023b7841b73" (UID: "33658ba2-9a0f-4edd-9559-7023b7841b73"). InnerVolumeSpecName "kube-api-access-pzmm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:22.060130 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.060000 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls" pod "33658ba2-9a0f-4edd-9559-7023b7841b73" (UID: "33658ba2-9a0f-4edd-9559-7023b7841b73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:22.160346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.160308 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:22.160346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.160337 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzmm5\" (UniqueName: \"kubernetes.io/projected/33658ba2-9a0f-4edd-9559-7023b7841b73-kube-api-access-pzmm5\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:22.160346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.160352 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33658ba2-9a0f-4edd-9559-7023b7841b73-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:22.634569 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.634530 5890 generic.go:356] "Generic (PLEG): container finished" podID="33658ba2-9a0f-4edd-9559-7023b7841b73" containerID="d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5" exitCode=0 Oct 31 01:27:22.634765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.634577 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" Oct 31 01:27:22.634765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.634603 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" event={"ID":"33658ba2-9a0f-4edd-9559-7023b7841b73","Type":"ContainerDied","Data":"d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5"} Oct 31 01:27:22.634765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.634650 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" event={"ID":"33658ba2-9a0f-4edd-9559-7023b7841b73","Type":"ContainerDied","Data":"749a31c88cddf70ca09c01814d693c91a7a0569035afda275d0cabd99c43163f"} Oct 31 01:27:22.634765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.634676 5890 scope.go:122] "RemoveContainer" containerID="d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5" Oct 31 01:27:22.647030 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.647007 5890 scope.go:122] "RemoveContainer" containerID="d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5" Oct 31 01:27:22.647332 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:22.647297 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5\": container with ID starting with d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5 not found: ID does not exist" containerID="d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5" Oct 31 01:27:22.647436 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.647342 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5"} err="failed to get container status \"d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5\": rpc error: code = NotFound desc = could not find container \"d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5\": container with ID starting with d1a2583946c4d575c2b390f747e1a7cd9ef53f1351aa9d184df2dea7bc15c9c5 not found: ID does not exist" Oct 31 01:27:22.772440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.772407 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll"] Oct 31 01:27:22.773131 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773111 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" Oct 31 01:27:22.773131 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773132 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773151 5890 cpu_manager.go:414] "containerMap: removing container" podUID="276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" containerName="payload-pre-processing" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773157 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" containerName="payload-pre-processing" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773172 5890 cpu_manager.go:414] "containerMap: removing container" podUID="746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" containerName="maas-api" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773186 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" containerName="maas-api" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773197 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773201 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773211 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e3629424-1787-406c-9bee-01f5fe05ff24" containerName="maas-api" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773216 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3629424-1787-406c-9bee-01f5fe05ff24" containerName="maas-api" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773238 5890 cpu_manager.go:414] "containerMap: removing container" podUID="33658ba2-9a0f-4edd-9559-7023b7841b73" containerName="maas-api" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773246 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33658ba2-9a0f-4edd-9559-7023b7841b73" containerName="maas-api" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773266 5890 cpu_manager.go:414] "containerMap: removing container" podUID="807f2910-69bb-4094-9027-5bf693b64fb3" containerName="payload-processing" Oct 31 01:27:22.773277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773272 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807f2910-69bb-4094-9027-5bf693b64fb3" containerName="payload-processing" Oct 31 01:27:22.773641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773366 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" Oct 31 01:27:22.773641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773376 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="276eb75d-7f2b-4451-bd5f-a7fc8ded55a7" containerName="payload-pre-processing" Oct 31 01:27:22.773641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773384 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="746ebe0d-c46c-4aa6-8f4a-c4c7ad26b24c" containerName="maas-api" Oct 31 01:27:22.773641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773392 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3629424-1787-406c-9bee-01f5fe05ff24" containerName="maas-api" Oct 31 01:27:22.773641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773403 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="33658ba2-9a0f-4edd-9559-7023b7841b73" containerName="maas-api" Oct 31 01:27:22.773641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.773413 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="807f2910-69bb-4094-9027-5bf693b64fb3" containerName="payload-processing" Oct 31 01:27:22.778303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.778283 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.802695 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.802666 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p"] Oct 31 01:27:22.818772 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.818740 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p"] Oct 31 01:27:22.828687 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.828655 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c61da9-openshift-default-dockercfg-56sw9\"" Oct 31 01:27:22.843617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.843591 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll"] Oct 31 01:27:22.868478 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.868445 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42be952b-15bf-41c7-b416-3f259d97849b-istio-podinfo\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.868644 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.868617 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-data\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.868702 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.868669 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbq86\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-kube-api-access-vbq86\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.868830 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.868806 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-certs\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.868967 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.868944 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-credential-socket\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.869021 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.869004 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-envoy\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.869086 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.869065 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-istio-token\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.869171 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.869155 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-socket\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.869216 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.869202 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42be952b-15bf-41c7-b416-3f259d97849b-istiod-ca-cert\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.970637 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-data\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.970696 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbq86\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-kube-api-access-vbq86\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971143 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-certs\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971253 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-credential-socket\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971278 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-data\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971322 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-envoy\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971348 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-istio-token\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971397 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-socket\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971441 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42be952b-15bf-41c7-b416-3f259d97849b-istiod-ca-cert\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971503 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42be952b-15bf-41c7-b416-3f259d97849b-istio-podinfo\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.971561 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-certs\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.972390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.972126 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-socket\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.973400 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.972854 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42be952b-15bf-41c7-b416-3f259d97849b-istiod-ca-cert\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.973400 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.973189 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-credential-socket\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.976075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.976053 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-envoy\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:22.976192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:22.976170 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42be952b-15bf-41c7-b416-3f259d97849b-istio-podinfo\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:23.030572 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.030538 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbq86\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-kube-api-access-vbq86\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:23.030761 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.030548 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-istio-token\") pod \"e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:23.060369 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.060332 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n"] Oct 31 01:27:23.060599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.060572 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" podUID="fc8301ab-23c3-4998-969c-153ff79dd3e4" containerName="maas-api" containerID="cri-o://638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22" gracePeriod=30 Oct 31 01:27:23.092702 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.092674 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:23.344328 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.344298 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:27:23.488593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.488552 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls\") pod \"fc8301ab-23c3-4998-969c-153ff79dd3e4\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " Oct 31 01:27:23.488813 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.488701 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8301ab-23c3-4998-969c-153ff79dd3e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8301ab-23c3-4998-969c-153ff79dd3e4-openshift-service-ca\") pod \"fc8301ab-23c3-4998-969c-153ff79dd3e4\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " Oct 31 01:27:23.489155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.489111 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8301ab-23c3-4998-969c-153ff79dd3e4-openshift-service-ca" pod "fc8301ab-23c3-4998-969c-153ff79dd3e4" (UID: "fc8301ab-23c3-4998-969c-153ff79dd3e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:23.489875 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.489452 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-metrics-certs\") pod \"fc8301ab-23c3-4998-969c-153ff79dd3e4\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " Oct 31 01:27:23.490019 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.489988 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8301ab-23c3-4998-969c-153ff79dd3e4-kube-api-access-kqssf\" (UniqueName: \"kubernetes.io/projected/fc8301ab-23c3-4998-969c-153ff79dd3e4-kube-api-access-kqssf\") pod \"fc8301ab-23c3-4998-969c-153ff79dd3e4\" (UID: \"fc8301ab-23c3-4998-969c-153ff79dd3e4\") " Oct 31 01:27:23.500362 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.500335 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8301ab-23c3-4998-969c-153ff79dd3e4-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:23.501291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.501264 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls" pod "fc8301ab-23c3-4998-969c-153ff79dd3e4" (UID: "fc8301ab-23c3-4998-969c-153ff79dd3e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:23.501291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.501276 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-metrics-certs" pod "fc8301ab-23c3-4998-969c-153ff79dd3e4" (UID: "fc8301ab-23c3-4998-969c-153ff79dd3e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:23.502691 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.502664 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8301ab-23c3-4998-969c-153ff79dd3e4-kube-api-access-kqssf" pod "fc8301ab-23c3-4998-969c-153ff79dd3e4" (UID: "fc8301ab-23c3-4998-969c-153ff79dd3e4"). InnerVolumeSpecName "kube-api-access-kqssf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:23.526097 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:27:23.526067 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42be952b_15bf_41c7_b416_3f259d97849b.slice/crio-beddc595869fd6e60998fe815a88aff2a3cd8d96880d103cad209a4317278e6f WatchSource:0}: Error finding container beddc595869fd6e60998fe815a88aff2a3cd8d96880d103cad209a4317278e6f: Status 404 returned error can't find the container with id beddc595869fd6e60998fe815a88aff2a3cd8d96880d103cad209a4317278e6f Oct 31 01:27:23.526508 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.526475 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll"] Oct 31 01:27:23.529155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.529113 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:27:23.529246 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.529201 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:27:23.529291 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.529245 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:27:23.602042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.601968 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:23.602206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.602045 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqssf\" (UniqueName: \"kubernetes.io/projected/fc8301ab-23c3-4998-969c-153ff79dd3e4-kube-api-access-kqssf\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:23.602206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.602084 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8301ab-23c3-4998-969c-153ff79dd3e4-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:23.646425 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.646384 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" event={"ID":"42be952b-15bf-41c7-b416-3f259d97849b","Type":"ContainerStarted","Data":"c027315a48fb1c90ab4423beb4d93f7fee0d632c1550f8b5b26470c6baae8659"} Oct 31 01:27:23.646631 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.646464 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" event={"ID":"42be952b-15bf-41c7-b416-3f259d97849b","Type":"ContainerStarted","Data":"beddc595869fd6e60998fe815a88aff2a3cd8d96880d103cad209a4317278e6f"} Oct 31 01:27:23.649868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.649832 5890 generic.go:356] "Generic (PLEG): container finished" podID="fc8301ab-23c3-4998-969c-153ff79dd3e4" containerID="638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22" exitCode=0 Oct 31 01:27:23.650025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.649881 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" Oct 31 01:27:23.650025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.649915 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" event={"ID":"fc8301ab-23c3-4998-969c-153ff79dd3e4","Type":"ContainerDied","Data":"638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22"} Oct 31 01:27:23.650025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.649959 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n" event={"ID":"fc8301ab-23c3-4998-969c-153ff79dd3e4","Type":"ContainerDied","Data":"514853a2cffa05e3fe5beeb27af308d47c6727530d88126c068ff343f608d068"} Oct 31 01:27:23.650025 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.649987 5890 scope.go:122] "RemoveContainer" containerID="638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22" Oct 31 01:27:23.672789 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.672767 5890 scope.go:122] "RemoveContainer" containerID="638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22" Oct 31 01:27:23.674229 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:23.674146 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22\": container with ID starting with 638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22 not found: ID does not exist" containerID="638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22" Oct 31 01:27:23.674229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.674189 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22"} err="failed to get container status \"638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22\": rpc error: code = NotFound desc = could not find container \"638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22\": container with ID starting with 638f2389261f6ac2f5c7ea8ec32c0bab6a9dd4ff81fa3898b09a28bdd2db5b22 not found: ID does not exist" Oct 31 01:27:23.768104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.768048 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podStartSLOduration=1.768032175 podStartE2EDuration="1.768032175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:23.765871328 +0000 UTC m=+3535.968121764" watchObservedRunningTime="2026-10-31 01:27:23.768032175 +0000 UTC m=+3535.970282613" Oct 31 01:27:23.874526 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.874443 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n"] Oct 31 01:27:23.879443 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.879416 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-08fcc4-6788f95788-gww8n"] Oct 31 01:27:23.935155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.935113 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33658ba2-9a0f-4edd-9559-7023b7841b73" path="/var/lib/kubelet/pods/33658ba2-9a0f-4edd-9559-7023b7841b73/volumes" Oct 31 01:27:23.935591 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:23.935577 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8301ab-23c3-4998-969c-153ff79dd3e4" path="/var/lib/kubelet/pods/fc8301ab-23c3-4998-969c-153ff79dd3e4/volumes" Oct 31 01:27:24.092967 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:24.092934 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:24.099551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:24.099524 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:24.660491 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:24.660462 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:24.661393 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:24.661368 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:27:24.914625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:24.914525 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb68dd-5cb59c7c65-f2t4p" podUID="33658ba2-9a0f-4edd-9559-7023b7841b73" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.168:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:27:25.470184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.470125 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:27:25.624231 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624193 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-envoy\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624267 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-kube-api-access-jh47n\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-kube-api-access-jh47n\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624295 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-token\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624320 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-credential-socket\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624405 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-podinfo\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624438 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a53953-ecea-4303-8be1-99b6f93bc3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a53953-ecea-4303-8be1-99b6f93bc3e9-istiod-ca-cert\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624479 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-data\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624565 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-socket\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624614 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-certs\") pod \"31a53953-ecea-4303-8be1-99b6f93bc3e9\" (UID: \"31a53953-ecea-4303-8be1-99b6f93bc3e9\") " Oct 31 01:27:25.624959 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.624933 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-data" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:25.625242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.625040 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a53953-ecea-4303-8be1-99b6f93bc3e9-istiod-ca-cert" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:25.625242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.625157 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-socket" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:25.625242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.625200 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-credential-socket" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:25.625460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.625300 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-certs" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:25.625460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.625308 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.625460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.625343 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.625460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.625361 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a53953-ecea-4303-8be1-99b6f93bc3e9-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.628108 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.627391 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-envoy" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:25.628108 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.627532 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-kube-api-access-jh47n" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "kube-api-access-jh47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:25.628108 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.627618 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-podinfo" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:27:25.628300 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.628214 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-token" pod "31a53953-ecea-4303-8be1-99b6f93bc3e9" (UID: "31a53953-ecea-4303-8be1-99b6f93bc3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:25.667755 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.667720 5890 generic.go:356] "Generic (PLEG): container finished" podID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerID="5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2" exitCode=0 Oct 31 01:27:25.667910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.667773 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" Oct 31 01:27:25.667910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.667810 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" event={"ID":"31a53953-ecea-4303-8be1-99b6f93bc3e9","Type":"ContainerDied","Data":"5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2"} Oct 31 01:27:25.667910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.667872 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr" event={"ID":"31a53953-ecea-4303-8be1-99b6f93bc3e9","Type":"ContainerDied","Data":"785cd89edafb4bc718973f6fc4d3157f14000f25d1562645829100232d615efc"} Oct 31 01:27:25.667910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.667903 5890 scope.go:122] "RemoveContainer" containerID="5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2" Oct 31 01:27:25.679309 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.679290 5890 scope.go:122] "RemoveContainer" containerID="446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b" Oct 31 01:27:25.688532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.688511 5890 scope.go:122] "RemoveContainer" containerID="5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2" Oct 31 01:27:25.688769 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:25.688746 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2\": container with ID starting with 5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2 not found: ID does not exist" containerID="5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2" Oct 31 01:27:25.688855 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.688779 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2"} err="failed to get container status \"5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2\": rpc error: code = NotFound desc = could not find container \"5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2\": container with ID starting with 5e199f6a3b0b2f89604d1fd5a6272e4deebe6423bbcae02a9a19bbdfa2dbcbb2 not found: ID does not exist" Oct 31 01:27:25.688855 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.688822 5890 scope.go:122] "RemoveContainer" containerID="446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b" Oct 31 01:27:25.689039 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:25.689023 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b\": container with ID starting with 446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b not found: ID does not exist" containerID="446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b" Oct 31 01:27:25.689078 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.689045 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b"} err="failed to get container status \"446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b\": rpc error: code = NotFound desc = could not find container \"446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b\": container with ID starting with 446f2a35a1cce5bbac4b6cf7a1cd515d9b0e4e949291e02a6b57858af77a7b3b not found: ID does not exist" Oct 31 01:27:25.714397 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.714375 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5"] Oct 31 01:27:25.719721 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.719699 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf"] Oct 31 01:27:25.727233 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.727176 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.727233 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.727201 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.727233 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.727218 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.727233 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.727232 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh47n\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-kube-api-access-jh47n\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.727410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.727246 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a53953-ecea-4303-8be1-99b6f93bc3e9-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.727410 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.727268 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a53953-ecea-4303-8be1-99b6f93bc3e9-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:25.738196 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:25.738146 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv"] Oct 31 01:27:26.056389 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:26.056296 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr"] Oct 31 01:27:26.060043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:26.060018 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-6b5091-openshift-default-776b46b7bf-sw4qr"] Oct 31 01:27:27.934638 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:27.934604 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" path="/var/lib/kubelet/pods/31a53953-ecea-4303-8be1-99b6f93bc3e9/volumes" Oct 31 01:27:30.672043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.671560 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9"] Oct 31 01:27:30.672548 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672421 5890 cpu_manager.go:414] "containerMap: removing container" podUID="fc8301ab-23c3-4998-969c-153ff79dd3e4" containerName="maas-api" Oct 31 01:27:30.672548 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672441 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8301ab-23c3-4998-969c-153ff79dd3e4" containerName="maas-api" Oct 31 01:27:30.672548 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672467 5890 cpu_manager.go:414] "containerMap: removing container" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" Oct 31 01:27:30.672548 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672475 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" Oct 31 01:27:30.672548 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672493 5890 cpu_manager.go:414] "containerMap: removing container" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" Oct 31 01:27:30.672548 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672500 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" Oct 31 01:27:30.672910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672680 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" Oct 31 01:27:30.672910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672694 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a53953-ecea-4303-8be1-99b6f93bc3e9" containerName="istio-proxy" Oct 31 01:27:30.672910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672710 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5016469-de22-402b-b73a-471dab674b25" containerName="istio-proxy" Oct 31 01:27:30.672910 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.672725 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8301ab-23c3-4998-969c-153ff79dd3e4" containerName="maas-api" Oct 31 01:27:30.678328 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.678303 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.730086 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.730049 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c61da9\"" Oct 31 01:27:30.795038 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.795000 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckf54\" (UniqueName: \"kubernetes.io/projected/e1879df4-734d-4002-9fff-4b5c1c1128e7-kube-api-access-ckf54\") pod \"payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.795367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.795341 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c61da9-dockercfg-g554s\"" Oct 31 01:27:30.795577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.795554 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1879df4-734d-4002-9fff-4b5c1c1128e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.799431 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.799396 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9"] Oct 31 01:27:30.826881 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.826844 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl"] Oct 31 01:27:30.831925 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.831900 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:30.878237 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.878205 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl"] Oct 31 01:27:30.897238 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.897197 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckf54\" (UniqueName: \"kubernetes.io/projected/e1879df4-734d-4002-9fff-4b5c1c1128e7-kube-api-access-ckf54\") pod \"payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.897420 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.897400 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1879df4-734d-4002-9fff-4b5c1c1128e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.898195 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.898174 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1879df4-734d-4002-9fff-4b5c1c1128e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.958476 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.958399 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckf54\" (UniqueName: \"kubernetes.io/projected/e1879df4-734d-4002-9fff-4b5c1c1128e7-kube-api-access-ckf54\") pod \"payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.993205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.993171 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:30.998944 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.998911 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68qx\" (UniqueName: \"kubernetes.io/projected/2a114a39-0af2-437e-8b9e-1353ae7b7267-kube-api-access-n68qx\") pod \"payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:30.999058 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:30.998979 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a114a39-0af2-437e-8b9e-1353ae7b7267-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:31.100804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.100426 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68qx\" (UniqueName: \"kubernetes.io/projected/2a114a39-0af2-437e-8b9e-1353ae7b7267-kube-api-access-n68qx\") pod \"payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:31.100804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.100477 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a114a39-0af2-437e-8b9e-1353ae7b7267-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:31.101243 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.101201 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a114a39-0af2-437e-8b9e-1353ae7b7267-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:31.162290 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.162249 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68qx\" (UniqueName: \"kubernetes.io/projected/2a114a39-0af2-437e-8b9e-1353ae7b7267-kube-api-access-n68qx\") pod \"payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:31.365242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.365204 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9"] Oct 31 01:27:31.367229 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:27:31.367199 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1879df4_734d_4002_9fff_4b5c1c1128e7.slice/crio-d2eb77911ba302a1d3bafcd332dfad53bbd45d0566b89de403162019c1c6ef3b WatchSource:0}: Error finding container d2eb77911ba302a1d3bafcd332dfad53bbd45d0566b89de403162019c1c6ef3b: Status 404 returned error can't find the container with id d2eb77911ba302a1d3bafcd332dfad53bbd45d0566b89de403162019c1c6ef3b Oct 31 01:27:31.451626 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.451597 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:31.709660 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.709619 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" event={"ID":"e1879df4-734d-4002-9fff-4b5c1c1128e7","Type":"ContainerStarted","Data":"d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8"} Oct 31 01:27:31.710139 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.709682 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" event={"ID":"e1879df4-734d-4002-9fff-4b5c1c1128e7","Type":"ContainerStarted","Data":"d2eb77911ba302a1d3bafcd332dfad53bbd45d0566b89de403162019c1c6ef3b"} Oct 31 01:27:31.710139 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.709815 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:31.824616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.824565 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" podStartSLOduration=1.8245526989999998 podStartE2EDuration="1.824552699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:31.823427135 +0000 UTC m=+3544.025677568" watchObservedRunningTime="2026-10-31 01:27:31.824552699 +0000 UTC m=+3544.026803131" Oct 31 01:27:31.831116 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:31.831083 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl"] Oct 31 01:27:31.832239 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:27:31.832217 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a114a39_0af2_437e_8b9e_1353ae7b7267.slice/crio-6694860fec8e0b558ad62cb9a9587dc493f83b4a6d675fae320a580589dc3a0b WatchSource:0}: Error finding container 6694860fec8e0b558ad62cb9a9587dc493f83b4a6d675fae320a580589dc3a0b: Status 404 returned error can't find the container with id 6694860fec8e0b558ad62cb9a9587dc493f83b4a6d675fae320a580589dc3a0b Oct 31 01:27:32.717748 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:32.717707 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" event={"ID":"2a114a39-0af2-437e-8b9e-1353ae7b7267","Type":"ContainerStarted","Data":"9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f"} Oct 31 01:27:32.718175 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:32.717777 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" event={"ID":"2a114a39-0af2-437e-8b9e-1353ae7b7267","Type":"ContainerStarted","Data":"6694860fec8e0b558ad62cb9a9587dc493f83b4a6d675fae320a580589dc3a0b"} Oct 31 01:27:32.718175 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:32.717921 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:32.830724 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:32.830673 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" podStartSLOduration=2.830661174 podStartE2EDuration="2.830661174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:32.828752551 +0000 UTC m=+3545.031002983" watchObservedRunningTime="2026-10-31 01:27:32.830661174 +0000 UTC m=+3545.032911607" Oct 31 01:27:35.223650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:35.223614 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc"] Oct 31 01:27:37.038281 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:37.038238 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:27:40.224060 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:40.224012 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" podUID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" containerName="payload-pre-processing" containerID="cri-o://edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437" gracePeriod=25 Oct 31 01:27:40.736214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:40.736170 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" containerID="cri-o://a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe" gracePeriod=45 Oct 31 01:27:40.740828 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:40.740706 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" containerID="cri-o://65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d" gracePeriod=45 Oct 31 01:27:40.758249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:40.758207 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" containerID="cri-o://4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e" gracePeriod=45 Oct 31 01:27:41.043546 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.043519 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 01:27:41.052694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.052664 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 01:27:41.083775 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.083738 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 01:27:41.127796 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.127749 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8584857a-893b-4637-beb1-b666e919ae56-tls-certs\" (UniqueName: \"kubernetes.io/secret/8584857a-893b-4637-beb1-b666e919ae56-tls-certs\") pod \"8584857a-893b-4637-beb1-b666e919ae56\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " Oct 31 01:27:41.127991 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.127956 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-model-cache\") pod \"4093e5df-ad6a-467e-a54a-c266fd6df904\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " Oct 31 01:27:41.128078 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.127996 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-model-cache\") pod \"8e1e960f-622b-4653-9725-962d26889964\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " Oct 31 01:27:41.128078 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128027 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-home\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-home\") pod \"8e1e960f-622b-4653-9725-962d26889964\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " Oct 31 01:27:41.128193 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128113 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-dshm\") pod \"8e1e960f-622b-4653-9725-962d26889964\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " Oct 31 01:27:41.128193 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128139 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-home\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-home\") pod \"8584857a-893b-4637-beb1-b666e919ae56\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " Oct 31 01:27:41.128193 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128189 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-tmp-dir\") pod \"8584857a-893b-4637-beb1-b666e919ae56\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " Oct 31 01:27:41.128322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128214 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-model-cache\") pod \"8584857a-893b-4637-beb1-b666e919ae56\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " Oct 31 01:27:41.128322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128264 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8584857a-893b-4637-beb1-b666e919ae56-kube-api-access-kg6mb\" (UniqueName: \"kubernetes.io/projected/8584857a-893b-4637-beb1-b666e919ae56-kube-api-access-kg6mb\") pod \"8584857a-893b-4637-beb1-b666e919ae56\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " Oct 31 01:27:41.128322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128292 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4093e5df-ad6a-467e-a54a-c266fd6df904-kube-api-access-z4m7j\" (UniqueName: \"kubernetes.io/projected/4093e5df-ad6a-467e-a54a-c266fd6df904-kube-api-access-z4m7j\") pod \"4093e5df-ad6a-467e-a54a-c266fd6df904\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " Oct 31 01:27:41.128434 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128321 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-tmp-dir\") pod \"4093e5df-ad6a-467e-a54a-c266fd6df904\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " Oct 31 01:27:41.128434 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128346 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-model-cache" pod "8e1e960f-622b-4653-9725-962d26889964" (UID: "8e1e960f-622b-4653-9725-962d26889964"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.128434 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128350 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-home\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-home\") pod \"4093e5df-ad6a-467e-a54a-c266fd6df904\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " Oct 31 01:27:41.128539 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128491 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-tmp-dir\") pod \"8e1e960f-622b-4653-9725-962d26889964\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " Oct 31 01:27:41.128588 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128562 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-home" pod "4093e5df-ad6a-467e-a54a-c266fd6df904" (UID: "4093e5df-ad6a-467e-a54a-c266fd6df904"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.128678 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128656 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4093e5df-ad6a-467e-a54a-c266fd6df904-tls-certs\" (UniqueName: \"kubernetes.io/secret/4093e5df-ad6a-467e-a54a-c266fd6df904-tls-certs\") pod \"4093e5df-ad6a-467e-a54a-c266fd6df904\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " Oct 31 01:27:41.128809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128696 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1e960f-622b-4653-9725-962d26889964-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1e960f-622b-4653-9725-962d26889964-tls-certs\") pod \"8e1e960f-622b-4653-9725-962d26889964\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " Oct 31 01:27:41.128809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128737 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-dshm\") pod \"8584857a-893b-4637-beb1-b666e919ae56\" (UID: \"8584857a-893b-4637-beb1-b666e919ae56\") " Oct 31 01:27:41.128809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128755 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-home" pod "8e1e960f-622b-4653-9725-962d26889964" (UID: "8e1e960f-622b-4653-9725-962d26889964"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.128809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128765 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1e960f-622b-4653-9725-962d26889964-kube-api-access-kgj6p\" (UniqueName: \"kubernetes.io/projected/8e1e960f-622b-4653-9725-962d26889964-kube-api-access-kgj6p\") pod \"8e1e960f-622b-4653-9725-962d26889964\" (UID: \"8e1e960f-622b-4653-9725-962d26889964\") " Oct 31 01:27:41.128990 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.128818 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-dshm\") pod \"4093e5df-ad6a-467e-a54a-c266fd6df904\" (UID: \"4093e5df-ad6a-467e-a54a-c266fd6df904\") " Oct 31 01:27:41.129367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.129340 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.129367 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.129361 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.129523 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.129376 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.129587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.129537 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-home" pod "8584857a-893b-4637-beb1-b666e919ae56" (UID: "8584857a-893b-4637-beb1-b666e919ae56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.129854 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.129750 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-tmp-dir" pod "8584857a-893b-4637-beb1-b666e919ae56" (UID: "8584857a-893b-4637-beb1-b666e919ae56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.130218 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.130191 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-tmp-dir" pod "4093e5df-ad6a-467e-a54a-c266fd6df904" (UID: "4093e5df-ad6a-467e-a54a-c266fd6df904"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.130816 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.130757 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-model-cache" pod "8584857a-893b-4637-beb1-b666e919ae56" (UID: "8584857a-893b-4637-beb1-b666e919ae56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.131028 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.131002 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-model-cache" pod "4093e5df-ad6a-467e-a54a-c266fd6df904" (UID: "4093e5df-ad6a-467e-a54a-c266fd6df904"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.131347 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.131322 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-tmp-dir" pod "8e1e960f-622b-4653-9725-962d26889964" (UID: "8e1e960f-622b-4653-9725-962d26889964"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.131718 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.131671 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-dshm" pod "8e1e960f-622b-4653-9725-962d26889964" (UID: "8e1e960f-622b-4653-9725-962d26889964"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.132207 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.132181 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4093e5df-ad6a-467e-a54a-c266fd6df904-kube-api-access-z4m7j" pod "4093e5df-ad6a-467e-a54a-c266fd6df904" (UID: "4093e5df-ad6a-467e-a54a-c266fd6df904"). InnerVolumeSpecName "kube-api-access-z4m7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:41.133185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.133133 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8584857a-893b-4637-beb1-b666e919ae56-tls-certs" pod "8584857a-893b-4637-beb1-b666e919ae56" (UID: "8584857a-893b-4637-beb1-b666e919ae56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:41.133185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.133153 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-dshm" pod "4093e5df-ad6a-467e-a54a-c266fd6df904" (UID: "4093e5df-ad6a-467e-a54a-c266fd6df904"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.133655 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.133619 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1e960f-622b-4653-9725-962d26889964-kube-api-access-kgj6p" pod "8e1e960f-622b-4653-9725-962d26889964" (UID: "8e1e960f-622b-4653-9725-962d26889964"). InnerVolumeSpecName "kube-api-access-kgj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:41.134138 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.134118 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8584857a-893b-4637-beb1-b666e919ae56-kube-api-access-kg6mb" pod "8584857a-893b-4637-beb1-b666e919ae56" (UID: "8584857a-893b-4637-beb1-b666e919ae56"). InnerVolumeSpecName "kube-api-access-kg6mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:41.134266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.134239 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1e960f-622b-4653-9725-962d26889964-tls-certs" pod "8e1e960f-622b-4653-9725-962d26889964" (UID: "8e1e960f-622b-4653-9725-962d26889964"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:41.134913 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.134873 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4093e5df-ad6a-467e-a54a-c266fd6df904-tls-certs" pod "4093e5df-ad6a-467e-a54a-c266fd6df904" (UID: "4093e5df-ad6a-467e-a54a-c266fd6df904"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:41.135545 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.135522 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-dshm" pod "8584857a-893b-4637-beb1-b666e919ae56" (UID: "8584857a-893b-4637-beb1-b666e919ae56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.230454 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230420 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230454 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230451 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4093e5df-ad6a-467e-a54a-c266fd6df904-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230454 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230461 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1e960f-622b-4653-9725-962d26889964-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230469 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230480 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgj6p\" (UniqueName: \"kubernetes.io/projected/8e1e960f-622b-4653-9725-962d26889964-kube-api-access-kgj6p\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230490 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230501 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8584857a-893b-4637-beb1-b666e919ae56-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230509 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230518 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e1e960f-622b-4653-9725-962d26889964-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230526 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230534 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230541 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8584857a-893b-4637-beb1-b666e919ae56-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230549 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg6mb\" (UniqueName: \"kubernetes.io/projected/8584857a-893b-4637-beb1-b666e919ae56-kube-api-access-kg6mb\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230559 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4m7j\" (UniqueName: \"kubernetes.io/projected/4093e5df-ad6a-467e-a54a-c266fd6df904-kube-api-access-z4m7j\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.230943 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.230567 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4093e5df-ad6a-467e-a54a-c266fd6df904-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.783532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.783499 5890 generic.go:356] "Generic (PLEG): container finished" podID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerID="65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d" exitCode=0 Oct 31 01:27:41.783760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.783548 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" Oct 31 01:27:41.783760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.783578 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" event={"ID":"4093e5df-ad6a-467e-a54a-c266fd6df904","Type":"ContainerDied","Data":"65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d"} Oct 31 01:27:41.783760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.783621 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf" event={"ID":"4093e5df-ad6a-467e-a54a-c266fd6df904","Type":"ContainerDied","Data":"ffa53682ab843fde2843ddf4eda0a10fa69865795bb5163f86698218772b77a0"} Oct 31 01:27:41.783760 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.783637 5890 scope.go:122] "RemoveContainer" containerID="65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d" Oct 31 01:27:41.785202 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.785154 5890 generic.go:356] "Generic (PLEG): container finished" podID="8584857a-893b-4637-beb1-b666e919ae56" containerID="4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e" exitCode=0 Oct 31 01:27:41.785292 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.785212 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" Oct 31 01:27:41.785292 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.785238 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" event={"ID":"8584857a-893b-4637-beb1-b666e919ae56","Type":"ContainerDied","Data":"4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e"} Oct 31 01:27:41.785426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.785294 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv" event={"ID":"8584857a-893b-4637-beb1-b666e919ae56","Type":"ContainerDied","Data":"b89de8718383103cd2a9460416be53c15a5e63735bbb9dbd5a243b97efd8e0c9"} Oct 31 01:27:41.786638 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.786618 5890 generic.go:356] "Generic (PLEG): container finished" podID="8e1e960f-622b-4653-9725-962d26889964" containerID="a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe" exitCode=0 Oct 31 01:27:41.786752 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.786662 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" Oct 31 01:27:41.786752 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.786693 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" event={"ID":"8e1e960f-622b-4653-9725-962d26889964","Type":"ContainerDied","Data":"a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe"} Oct 31 01:27:41.786752 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.786722 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5" event={"ID":"8e1e960f-622b-4653-9725-962d26889964","Type":"ContainerDied","Data":"d8a419d11e20829ac1e2355ba6787d294c81c79434dafc6581c19b0286708b0d"} Oct 31 01:27:41.795469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.795160 5890 scope.go:122] "RemoveContainer" containerID="65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d" Oct 31 01:27:41.796012 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:41.795986 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d\": container with ID starting with 65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d not found: ID does not exist" containerID="65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d" Oct 31 01:27:41.796079 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.796029 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d"} err="failed to get container status \"65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d\": rpc error: code = NotFound desc = could not find container \"65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d\": container with ID starting with 65da961835cab23472804ee3442aa5f30d85cac8500415d15a02475b8abec96d not found: ID does not exist" Oct 31 01:27:41.796079 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.796052 5890 scope.go:122] "RemoveContainer" containerID="4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e" Oct 31 01:27:41.807316 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.807297 5890 scope.go:122] "RemoveContainer" containerID="4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e" Oct 31 01:27:41.807568 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:41.807552 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e\": container with ID starting with 4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e not found: ID does not exist" containerID="4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e" Oct 31 01:27:41.807631 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.807575 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e"} err="failed to get container status \"4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e\": rpc error: code = NotFound desc = could not find container \"4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e\": container with ID starting with 4c5071e09ddfefcb24642db16fae2c627ad3d6aa6c4e5bba91bbe413dbabb45e not found: ID does not exist" Oct 31 01:27:41.807631 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.807590 5890 scope.go:122] "RemoveContainer" containerID="a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe" Oct 31 01:27:41.816639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.816621 5890 scope.go:122] "RemoveContainer" containerID="a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe" Oct 31 01:27:41.816953 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:27:41.816932 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe\": container with ID starting with a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe not found: ID does not exist" containerID="a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe" Oct 31 01:27:41.817028 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.816961 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe"} err="failed to get container status \"a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe\": rpc error: code = NotFound desc = could not find container \"a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe\": container with ID starting with a30f00889736f067dec754a407256d25ec3c0f5548059e8495f43de906991bfe not found: ID does not exist" Oct 31 01:27:41.941948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.941916 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf"] Oct 31 01:27:41.945694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:41.945669 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-w3-dd98f0-kserve-845cf54849-b29nf"] Oct 31 01:27:42.050714 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:42.050626 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv"] Oct 31 01:27:42.053577 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:42.053548 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/simulator-w3-dd98f0-kserve-c467f8d55-x94mv"] Oct 31 01:27:42.159917 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:42.159883 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5"] Oct 31 01:27:42.165806 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:42.165754 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/distinct-2-w3-dd98f0-kserve-754859678b-dtcq5"] Oct 31 01:27:42.721190 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:42.721160 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:27:43.143112 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:43.143013 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" podUID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:27:43.727743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:43.727707 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:27:43.933498 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:43.933467 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" path="/var/lib/kubelet/pods/4093e5df-ad6a-467e-a54a-c266fd6df904/volumes" Oct 31 01:27:43.933969 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:43.933954 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8584857a-893b-4637-beb1-b666e919ae56" path="/var/lib/kubelet/pods/8584857a-893b-4637-beb1-b666e919ae56/volumes" Oct 31 01:27:43.934378 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:43.934366 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1e960f-622b-4653-9725-962d26889964" path="/var/lib/kubelet/pods/8e1e960f-622b-4653-9725-962d26889964/volumes" Oct 31 01:27:53.143187 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:27:53.143152 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" podUID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:00.492965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.492906 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 01:28:00.676610 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.676575 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075ca02a-e1be-4229-997a-f1bdc5fbcb31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075ca02a-e1be-4229-997a-f1bdc5fbcb31-plugins-config-volume\") pod \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " Oct 31 01:28:00.676773 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.676623 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075ca02a-e1be-4229-997a-f1bdc5fbcb31-kube-api-access-dnxnc\" (UniqueName: \"kubernetes.io/projected/075ca02a-e1be-4229-997a-f1bdc5fbcb31-kube-api-access-dnxnc\") pod \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\" (UID: \"075ca02a-e1be-4229-997a-f1bdc5fbcb31\") " Oct 31 01:28:00.677099 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.677073 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075ca02a-e1be-4229-997a-f1bdc5fbcb31-plugins-config-volume" pod "075ca02a-e1be-4229-997a-f1bdc5fbcb31" (UID: "075ca02a-e1be-4229-997a-f1bdc5fbcb31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:00.678817 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.678776 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075ca02a-e1be-4229-997a-f1bdc5fbcb31-kube-api-access-dnxnc" pod "075ca02a-e1be-4229-997a-f1bdc5fbcb31" (UID: "075ca02a-e1be-4229-997a-f1bdc5fbcb31"). InnerVolumeSpecName "kube-api-access-dnxnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:00.778678 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.778595 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075ca02a-e1be-4229-997a-f1bdc5fbcb31-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:00.778678 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.778626 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnxnc\" (UniqueName: \"kubernetes.io/projected/075ca02a-e1be-4229-997a-f1bdc5fbcb31-kube-api-access-dnxnc\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:00.919026 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.918985 5890 generic.go:356] "Generic (PLEG): container finished" podID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" containerID="edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437" exitCode=0 Oct 31 01:28:00.919257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.919032 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" Oct 31 01:28:00.919257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.919073 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" event={"ID":"075ca02a-e1be-4229-997a-f1bdc5fbcb31","Type":"ContainerDied","Data":"edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437"} Oct 31 01:28:00.919257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.919145 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc" event={"ID":"075ca02a-e1be-4229-997a-f1bdc5fbcb31","Type":"ContainerDied","Data":"9ce9165ca1a400b533c67190ea42898aff08a3d3aaf93d8f575dd8ed2aa24dcb"} Oct 31 01:28:00.919257 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.919177 5890 scope.go:122] "RemoveContainer" containerID="edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437" Oct 31 01:28:00.929483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.929462 5890 scope.go:122] "RemoveContainer" containerID="edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437" Oct 31 01:28:00.929748 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:00.929726 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437\": container with ID starting with edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437 not found: ID does not exist" containerID="edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437" Oct 31 01:28:00.929873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:00.929762 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437"} err="failed to get container status \"edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437\": rpc error: code = NotFound desc = could not find container \"edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437\": container with ID starting with edf75ceedaa7892c198fa617434bb986a052be9630a480cb87d78a7cacdad437 not found: ID does not exist" Oct 31 01:28:01.076698 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:01.076609 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc"] Oct 31 01:28:01.082336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:01.082306 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd98f0-9b775c64c-4f9xc"] Oct 31 01:28:01.933185 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:01.933150 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" path="/var/lib/kubelet/pods/075ca02a-e1be-4229-997a-f1bdc5fbcb31/volumes" Oct 31 01:28:07.018299 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:07.018262 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:28:26.550170 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.550132 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc"] Oct 31 01:28:26.550856 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.550357 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" podUID="28e641e9-efa8-4b1d-85e6-53f58407efb8" containerName="maas-api" containerID="cri-o://f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458" gracePeriod=30 Oct 31 01:28:26.819547 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.819520 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:28:26.993296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.993260 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-maas-api-tls\") pod \"28e641e9-efa8-4b1d-85e6-53f58407efb8\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " Oct 31 01:28:26.993483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.993310 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-metrics-certs\") pod \"28e641e9-efa8-4b1d-85e6-53f58407efb8\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " Oct 31 01:28:26.993483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.993356 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e641e9-efa8-4b1d-85e6-53f58407efb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e641e9-efa8-4b1d-85e6-53f58407efb8-openshift-service-ca\") pod \"28e641e9-efa8-4b1d-85e6-53f58407efb8\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " Oct 31 01:28:26.993483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.993445 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e641e9-efa8-4b1d-85e6-53f58407efb8-kube-api-access-2rvl9\" (UniqueName: \"kubernetes.io/projected/28e641e9-efa8-4b1d-85e6-53f58407efb8-kube-api-access-2rvl9\") pod \"28e641e9-efa8-4b1d-85e6-53f58407efb8\" (UID: \"28e641e9-efa8-4b1d-85e6-53f58407efb8\") " Oct 31 01:28:26.993892 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.993866 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e641e9-efa8-4b1d-85e6-53f58407efb8-openshift-service-ca" pod "28e641e9-efa8-4b1d-85e6-53f58407efb8" (UID: "28e641e9-efa8-4b1d-85e6-53f58407efb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:26.995731 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.995706 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-metrics-certs" pod "28e641e9-efa8-4b1d-85e6-53f58407efb8" (UID: "28e641e9-efa8-4b1d-85e6-53f58407efb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:26.995885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.995716 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-maas-api-tls" pod "28e641e9-efa8-4b1d-85e6-53f58407efb8" (UID: "28e641e9-efa8-4b1d-85e6-53f58407efb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:26.995885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:26.995820 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e641e9-efa8-4b1d-85e6-53f58407efb8-kube-api-access-2rvl9" pod "28e641e9-efa8-4b1d-85e6-53f58407efb8" (UID: "28e641e9-efa8-4b1d-85e6-53f58407efb8"). InnerVolumeSpecName "kube-api-access-2rvl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:27.095549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.095469 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:27.095549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.095496 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28e641e9-efa8-4b1d-85e6-53f58407efb8-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:27.095549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.095507 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e641e9-efa8-4b1d-85e6-53f58407efb8-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:27.095549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.095515 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvl9\" (UniqueName: \"kubernetes.io/projected/28e641e9-efa8-4b1d-85e6-53f58407efb8-kube-api-access-2rvl9\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:27.115277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.115246 5890 generic.go:356] "Generic (PLEG): container finished" podID="28e641e9-efa8-4b1d-85e6-53f58407efb8" containerID="f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458" exitCode=0 Oct 31 01:28:27.115450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.115290 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" Oct 31 01:28:27.115450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.115331 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" event={"ID":"28e641e9-efa8-4b1d-85e6-53f58407efb8","Type":"ContainerDied","Data":"f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458"} Oct 31 01:28:27.115450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.115389 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc" event={"ID":"28e641e9-efa8-4b1d-85e6-53f58407efb8","Type":"ContainerDied","Data":"694f1f32ae610164dbb89d4b7cc3e5d5500f5058ca925bb199bc655083c73d69"} Oct 31 01:28:27.115450 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.115418 5890 scope.go:122] "RemoveContainer" containerID="f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458" Oct 31 01:28:27.126629 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.126599 5890 scope.go:122] "RemoveContainer" containerID="f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458" Oct 31 01:28:27.126982 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:27.126960 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458\": container with ID starting with f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458 not found: ID does not exist" containerID="f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458" Oct 31 01:28:27.127067 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.126990 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458"} err="failed to get container status \"f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458\": rpc error: code = NotFound desc = could not find container \"f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458\": container with ID starting with f13f2a5b7ca86a98e3f460a2a247469ec28b0ebda96defebfa5421072dc41458 not found: ID does not exist" Oct 31 01:28:27.274003 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.273969 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc"] Oct 31 01:28:27.278877 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.278851 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4397fc-8cdd75f6d-z56zc"] Oct 31 01:28:27.935318 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:27.935285 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e641e9-efa8-4b1d-85e6-53f58407efb8" path="/var/lib/kubelet/pods/28e641e9-efa8-4b1d-85e6-53f58407efb8/volumes" Oct 31 01:28:29.037752 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:29.037721 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:28:29.149522 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:29.149496 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:28:33.455427 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.455386 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg"] Oct 31 01:28:33.456048 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456029 5890 cpu_manager.go:414] "containerMap: removing container" podUID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" containerName="payload-pre-processing" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456051 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" containerName="payload-pre-processing" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456075 5890 cpu_manager.go:414] "containerMap: removing container" podUID="28e641e9-efa8-4b1d-85e6-53f58407efb8" containerName="maas-api" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456080 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e641e9-efa8-4b1d-85e6-53f58407efb8" containerName="maas-api" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456119 5890 cpu_manager.go:414] "containerMap: removing container" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456125 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456138 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456146 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" Oct 31 01:28:33.456153 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456154 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" Oct 31 01:28:33.456435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456160 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" Oct 31 01:28:33.456435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456279 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="4093e5df-ad6a-467e-a54a-c266fd6df904" containerName="main" Oct 31 01:28:33.456435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456291 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8584857a-893b-4637-beb1-b666e919ae56" containerName="main" Oct 31 01:28:33.456435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456299 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1e960f-622b-4653-9725-962d26889964" containerName="main" Oct 31 01:28:33.456435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456311 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="075ca02a-e1be-4229-997a-f1bdc5fbcb31" containerName="payload-pre-processing" Oct 31 01:28:33.456435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.456319 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e641e9-efa8-4b1d-85e6-53f58407efb8" containerName="maas-api" Oct 31 01:28:33.459960 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.459940 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.474855 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.474830 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-metrics-certs\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.474966 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.474875 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.475073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.474991 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8n2\" (UniqueName: \"kubernetes.io/projected/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-kube-api-access-pj8n2\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.475133 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.475071 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.510131 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.510104 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c61da9-serving-cert\"" Oct 31 01:28:33.560340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.560310 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg"] Oct 31 01:28:33.575871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.575828 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-metrics-certs\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.576045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.575916 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.576045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.576025 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8n2\" (UniqueName: \"kubernetes.io/projected/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-kube-api-access-pj8n2\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.576174 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.576133 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.576255 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:33.576235 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-serving-cert: secret "maas-api-e2e-extmod-a-c61da9-serving-cert" not found Oct 31 01:28:33.576320 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:33.576304 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls podName:9b545ee4-8ec4-4f82-8e1e-182d95a48bc1 nodeName:}" failed. No retries permitted until 2026-10-31 01:28:34.076288954 +0000 UTC m=+3606.278539377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls") pod "maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" (UID: "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1") : secret "maas-api-e2e-extmod-a-c61da9-serving-cert" not found Oct 31 01:28:33.576737 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.576713 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.578639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.578617 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-metrics-certs\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:33.635184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:33.635153 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8n2\" (UniqueName: \"kubernetes.io/projected/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-kube-api-access-pj8n2\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:34.083125 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:34.083081 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:34.085759 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:34.085732 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:34.374379 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:34.374291 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:34.750149 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:34.750122 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg"] Oct 31 01:28:34.752125 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:28:34.752086 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b545ee4_8ec4_4f82_8e1e_182d95a48bc1.slice/crio-1664da182994745b7a8d53d4d1c65593558ed1d584bd9e3075295a7dc0038692 WatchSource:0}: Error finding container 1664da182994745b7a8d53d4d1c65593558ed1d584bd9e3075295a7dc0038692: Status 404 returned error can't find the container with id 1664da182994745b7a8d53d4d1c65593558ed1d584bd9e3075295a7dc0038692 Oct 31 01:28:34.753651 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:34.753631 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:28:35.180057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:35.180026 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" event={"ID":"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1","Type":"ContainerStarted","Data":"1664da182994745b7a8d53d4d1c65593558ed1d584bd9e3075295a7dc0038692"} Oct 31 01:28:36.189373 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:36.189333 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" event={"ID":"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1","Type":"ContainerStarted","Data":"efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9"} Oct 31 01:28:36.189814 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:36.189465 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:36.328451 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:36.328400 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" podStartSLOduration=2.3227910290000002 podStartE2EDuration="3.328384951s" totalImagesPullingTime="1.005593922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:36.32437542 +0000 UTC m=+3608.526625852" watchObservedRunningTime="2026-10-31 01:28:36.328384951 +0000 UTC m=+3608.530635384" Oct 31 01:28:37.019507 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:37.019461 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:28:42.201851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:42.201820 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:28:52.548532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:52.548491 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd"] Oct 31 01:28:52.549471 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:52.549433 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" containerID="cri-o://5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b" gracePeriod=30 Oct 31 01:28:54.522544 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:54.522506 5890 patch_prober.go:29] interesting pod/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:28:54.522976 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:54.522562 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:28:57.830762 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.830707 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:28:57.878100 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878073 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-data\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878280 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878133 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-certs\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878280 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878192 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-envoy\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878280 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878243 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istiod-ca-cert\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878280 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878267 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-podinfo\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878289 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-kube-api-access-gppj2\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-kube-api-access-gppj2\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878357 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-credential-socket\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878368 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-data" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:57.878519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878411 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-token\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-token\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878458 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-certs" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:57.878519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878479 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-socket\") pod \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\" (UID: \"6d45daef-a5b8-4c2f-9b19-9e74a27288e2\") " Oct 31 01:28:57.878873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878709 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-credential-socket" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:57.878873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878739 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istiod-ca-cert" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:57.878979 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.878921 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-socket" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:57.879130 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.879073 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.879130 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.879108 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.879130 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.879123 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.879264 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.879137 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.879264 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.879152 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.880715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.880682 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-podinfo" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:28:57.880999 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.880977 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-token" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:57.881099 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.881077 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-envoy" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:57.881156 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.881103 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-kube-api-access-gppj2" pod "6d45daef-a5b8-4c2f-9b19-9e74a27288e2" (UID: "6d45daef-a5b8-4c2f-9b19-9e74a27288e2"). InnerVolumeSpecName "kube-api-access-gppj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:57.980087 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.980052 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.980087 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.980087 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.980307 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.980101 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gppj2\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-kube-api-access-gppj2\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.980307 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:57.980116 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d45daef-a5b8-4c2f-9b19-9e74a27288e2-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:58.367416 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.367378 5890 generic.go:356] "Generic (PLEG): container finished" podID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerID="5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b" exitCode=0 Oct 31 01:28:58.367599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.367583 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" event={"ID":"6d45daef-a5b8-4c2f-9b19-9e74a27288e2","Type":"ContainerDied","Data":"5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b"} Oct 31 01:28:58.367652 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.367638 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" event={"ID":"6d45daef-a5b8-4c2f-9b19-9e74a27288e2","Type":"ContainerDied","Data":"ef93e9a6559b7b2a781c8c1e274ebe6cf44a254607c3c30b3002bc8b3376df67"} Oct 31 01:28:58.367689 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.367668 5890 scope.go:122] "RemoveContainer" containerID="5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b" Oct 31 01:28:58.367868 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.367855 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd" Oct 31 01:28:58.379362 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.379340 5890 scope.go:122] "RemoveContainer" containerID="1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699" Oct 31 01:28:58.389395 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.389374 5890 scope.go:122] "RemoveContainer" containerID="5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b" Oct 31 01:28:58.389666 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:58.389646 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b\": container with ID starting with 5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b not found: ID does not exist" containerID="5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b" Oct 31 01:28:58.389735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.389680 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b"} err="failed to get container status \"5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b\": rpc error: code = NotFound desc = could not find container \"5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b\": container with ID starting with 5d930e310f8a8bc1a151e227bc02005576efc9d5791f5cac2fdb5bf70903c37b not found: ID does not exist" Oct 31 01:28:58.389735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.389702 5890 scope.go:122] "RemoveContainer" containerID="1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699" Oct 31 01:28:58.390019 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:28:58.389999 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699\": container with ID starting with 1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699 not found: ID does not exist" containerID="1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699" Oct 31 01:28:58.390081 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.390025 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699"} err="failed to get container status \"1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699\": rpc error: code = NotFound desc = could not find container \"1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699\": container with ID starting with 1edb06307a1bba231c90cc6dd0c611930a2f2d515815ba60cecffc98bb36e699 not found: ID does not exist" Oct 31 01:28:58.529952 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.529912 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd"] Oct 31 01:28:58.536340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:58.536302 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-9332a5-openshift-default-6556fcb9df-f2hqd"] Oct 31 01:28:59.932592 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:28:59.932548 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" path="/var/lib/kubelet/pods/6d45daef-a5b8-4c2f-9b19-9e74a27288e2/volumes" Oct 31 01:29:07.018239 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:07.018194 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:29:25.898325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:25.898288 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb"] Oct 31 01:29:25.898751 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:25.898511 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" podUID="8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" containerName="maas-api" containerID="cri-o://4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143" gracePeriod=30 Oct 31 01:29:26.178083 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.178057 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 01:29:26.218975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.218945 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-kube-api-access-b7zkr\" (UniqueName: \"kubernetes.io/projected/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-kube-api-access-b7zkr\") pod \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " Oct 31 01:29:26.219128 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.219007 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-metrics-certs\") pod \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " Oct 31 01:29:26.219181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.219133 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-openshift-service-ca\") pod \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " Oct 31 01:29:26.219181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.219170 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls\") pod \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\" (UID: \"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8\") " Oct 31 01:29:26.219490 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.219459 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-openshift-service-ca" pod "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" (UID: "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:26.219634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.219601 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:26.221167 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.221141 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-metrics-certs" pod "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" (UID: "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:26.221311 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.221254 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls" pod "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" (UID: "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:26.221311 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.221288 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-kube-api-access-b7zkr" pod "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" (UID: "8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8"). InnerVolumeSpecName "kube-api-access-b7zkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:26.320930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.320893 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zkr\" (UniqueName: \"kubernetes.io/projected/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-kube-api-access-b7zkr\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:26.320930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.320925 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:26.320930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.320936 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:26.564697 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.564660 5890 generic.go:356] "Generic (PLEG): container finished" podID="8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" containerID="4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143" exitCode=0 Oct 31 01:29:26.564930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.564710 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" Oct 31 01:29:26.564930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.564745 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" event={"ID":"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8","Type":"ContainerDied","Data":"4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143"} Oct 31 01:29:26.564930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.564834 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb" event={"ID":"8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8","Type":"ContainerDied","Data":"cc370edbc0b2550381df1850ba2a4130de23d6b01a7848acda36cfbcd1f7db70"} Oct 31 01:29:26.564930 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.564865 5890 scope.go:122] "RemoveContainer" containerID="4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143" Oct 31 01:29:26.576495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.576475 5890 scope.go:122] "RemoveContainer" containerID="4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143" Oct 31 01:29:26.576799 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:26.576767 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143\": container with ID starting with 4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143 not found: ID does not exist" containerID="4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143" Oct 31 01:29:26.576908 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.576820 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143"} err="failed to get container status \"4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143\": rpc error: code = NotFound desc = could not find container \"4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143\": container with ID starting with 4df243a6b77c17e78ebc0f801922e59dcdc2dae7ae6f8d385de0562661763143 not found: ID does not exist" Oct 31 01:29:26.723362 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.723325 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb"] Oct 31 01:29:26.729094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:26.729059 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd98f0-65cc854f57-8d8bb"] Oct 31 01:29:27.933473 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:27.933441 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" path="/var/lib/kubelet/pods/8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8/volumes" Oct 31 01:29:29.393028 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:29.392991 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj"] Oct 31 01:29:29.393427 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:29.393280 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" containerID="cri-o://3781eb0cf69af4a3ccee5f574463d1075a3aac99a12b1fc2e1d00d4494e5c5b2" gracePeriod=30 Oct 31 01:29:33.394807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.394035 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt"] Oct 31 01:29:33.394807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.394288 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" podUID="be4c13ff-9583-4f2e-909e-549e1286ea06" containerName="maas-api" containerID="cri-o://331e718dda6ef34947c81d961baf3b33031535d6c1fb4e2e5d4dbdf21a227613" gracePeriod=30 Oct 31 01:29:33.619312 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.619268 5890 generic.go:356] "Generic (PLEG): container finished" podID="be4c13ff-9583-4f2e-909e-549e1286ea06" containerID="331e718dda6ef34947c81d961baf3b33031535d6c1fb4e2e5d4dbdf21a227613" exitCode=0 Oct 31 01:29:33.619496 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.619372 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" event={"ID":"be4c13ff-9583-4f2e-909e-549e1286ea06","Type":"ContainerDied","Data":"331e718dda6ef34947c81d961baf3b33031535d6c1fb4e2e5d4dbdf21a227613"} Oct 31 01:29:33.693221 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.693106 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:29:33.814398 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.814343 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-metrics-certs\") pod \"be4c13ff-9583-4f2e-909e-549e1286ea06\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " Oct 31 01:29:33.814592 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.814444 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4c13ff-9583-4f2e-909e-549e1286ea06-kube-api-access-qtwnj\" (UniqueName: \"kubernetes.io/projected/be4c13ff-9583-4f2e-909e-549e1286ea06-kube-api-access-qtwnj\") pod \"be4c13ff-9583-4f2e-909e-549e1286ea06\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " Oct 31 01:29:33.814592 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.814469 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls\") pod \"be4c13ff-9583-4f2e-909e-549e1286ea06\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " Oct 31 01:29:33.814720 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.814642 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4c13ff-9583-4f2e-909e-549e1286ea06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4c13ff-9583-4f2e-909e-549e1286ea06-openshift-service-ca\") pod \"be4c13ff-9583-4f2e-909e-549e1286ea06\" (UID: \"be4c13ff-9583-4f2e-909e-549e1286ea06\") " Oct 31 01:29:33.815505 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.815471 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4c13ff-9583-4f2e-909e-549e1286ea06-openshift-service-ca" pod "be4c13ff-9583-4f2e-909e-549e1286ea06" (UID: "be4c13ff-9583-4f2e-909e-549e1286ea06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:33.817979 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.817619 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-metrics-certs" pod "be4c13ff-9583-4f2e-909e-549e1286ea06" (UID: "be4c13ff-9583-4f2e-909e-549e1286ea06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:33.818190 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.818165 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls" pod "be4c13ff-9583-4f2e-909e-549e1286ea06" (UID: "be4c13ff-9583-4f2e-909e-549e1286ea06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:33.818271 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.818178 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4c13ff-9583-4f2e-909e-549e1286ea06-kube-api-access-qtwnj" pod "be4c13ff-9583-4f2e-909e-549e1286ea06" (UID: "be4c13ff-9583-4f2e-909e-549e1286ea06"). InnerVolumeSpecName "kube-api-access-qtwnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:33.916272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.916190 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:33.916272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.916219 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwnj\" (UniqueName: \"kubernetes.io/projected/be4c13ff-9583-4f2e-909e-549e1286ea06-kube-api-access-qtwnj\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:33.916272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.916229 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4c13ff-9583-4f2e-909e-549e1286ea06-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:33.916272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:33.916238 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4c13ff-9583-4f2e-909e-549e1286ea06-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.628387 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.628353 5890 generic.go:356] "Generic (PLEG): container finished" podID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerID="3781eb0cf69af4a3ccee5f574463d1075a3aac99a12b1fc2e1d00d4494e5c5b2" exitCode=0 Oct 31 01:29:34.628875 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.628493 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" event={"ID":"1a9096a8-7cf2-4454-a1c3-73cea55b63fc","Type":"ContainerDied","Data":"3781eb0cf69af4a3ccee5f574463d1075a3aac99a12b1fc2e1d00d4494e5c5b2"} Oct 31 01:29:34.628875 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.628572 5890 scope.go:122] "RemoveContainer" containerID="ff65640f1bede847685b9430dc25365b9efea422c10fc9a54b5a99e3536fb5dc" Oct 31 01:29:34.631504 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.631472 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" event={"ID":"be4c13ff-9583-4f2e-909e-549e1286ea06","Type":"ContainerDied","Data":"f738616a34fcdcbfc5326cbc81d4e61b6c93723196fcf4f73a25fb4d7d59515d"} Oct 31 01:29:34.631630 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.631485 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt" Oct 31 01:29:34.691363 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.691337 5890 scope.go:122] "RemoveContainer" containerID="331e718dda6ef34947c81d961baf3b33031535d6c1fb4e2e5d4dbdf21a227613" Oct 31 01:29:34.731300 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.731275 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:29:34.801965 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.801929 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt"] Oct 31 01:29:34.827401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827361 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-certs\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827411 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istiod-ca-cert\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827439 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-podinfo\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827481 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-kube-api-access-bxbgh\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-kube-api-access-bxbgh\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827503 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-envoy\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827846 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827584 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-token\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-token\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827846 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827615 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-socket\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827846 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827675 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-data\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.827846 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827680 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-certs" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:34.828055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827959 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-socket" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:34.828055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.827995 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istiod-ca-cert" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:34.828173 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.828137 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-credential-socket\") pod \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\" (UID: \"1a9096a8-7cf2-4454-a1c3-73cea55b63fc\") " Oct 31 01:29:34.828412 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.828389 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-credential-socket" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:34.828739 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.828719 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.828844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.828745 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.828844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.828759 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.828844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.828780 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.828844 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.828718 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-data" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:34.829987 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.829958 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-envoy" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:34.830195 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.830172 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-token" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:34.830390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.830371 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-podinfo" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:29:34.830456 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.830404 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-kube-api-access-bxbgh" pod "1a9096a8-7cf2-4454-a1c3-73cea55b63fc" (UID: "1a9096a8-7cf2-4454-a1c3-73cea55b63fc"). InnerVolumeSpecName "kube-api-access-bxbgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:34.867810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.867284 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8fb7f0-84949c859f-tm9dt"] Oct 31 01:29:34.931804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.930629 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.931804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.930669 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxbgh\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-kube-api-access-bxbgh\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.931804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.930684 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.931804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.930697 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.931804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.930711 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9096a8-7cf2-4454-a1c3-73cea55b63fc-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.955855 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.954828 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6"] Oct 31 01:29:34.967253 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:34.967204 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz"] Oct 31 01:29:35.641656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.641618 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" event={"ID":"1a9096a8-7cf2-4454-a1c3-73cea55b63fc","Type":"ContainerDied","Data":"584b60d4bf0b4be762f75c4bd543ab1962eb45a6f53073693b077ca4f288dcad"} Oct 31 01:29:35.642172 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.641678 5890 scope.go:122] "RemoveContainer" containerID="3781eb0cf69af4a3ccee5f574463d1075a3aac99a12b1fc2e1d00d4494e5c5b2" Oct 31 01:29:35.642172 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.641814 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj" Oct 31 01:29:35.676296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.676260 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp"] Oct 31 01:29:35.676521 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.676500 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" containerID="cri-o://aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57" gracePeriod=30 Oct 31 01:29:35.933895 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.933759 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4c13ff-9583-4f2e-909e-549e1286ea06" path="/var/lib/kubelet/pods/be4c13ff-9583-4f2e-909e-549e1286ea06/volumes" Oct 31 01:29:35.934265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.934244 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj"] Oct 31 01:29:35.934265 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:35.934294 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3be6fa-openshift-default-7b46d4f477-plkgj"] Oct 31 01:29:37.018738 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:37.018696 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:29:37.934962 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:37.934929 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" path="/var/lib/kubelet/pods/1a9096a8-7cf2-4454-a1c3-73cea55b63fc/volumes" Oct 31 01:29:40.477523 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:40.477485 5890 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:29:40.478339 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:40.478308 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:29:40.701951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:40.701919 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:29:40.702111 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:40.702004 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:29:40.989735 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:40.989699 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 01:29:41.132525 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132442 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-data\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.132525 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132489 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-socket\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.132812 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132546 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-kube-api-access-8rxt5\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-kube-api-access-8rxt5\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.132812 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132602 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-token\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-token\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.132812 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132650 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467d4202-b55f-4b78-8f8c-07184fbb79b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467d4202-b55f-4b78-8f8c-07184fbb79b1-istiod-ca-cert\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.132812 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132728 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-data" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:41.133080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132842 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-socket" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:41.133080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.132953 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-certs\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.133080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133016 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467d4202-b55f-4b78-8f8c-07184fbb79b1-istiod-ca-cert" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:41.133080 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133020 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-envoy\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.133301 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133125 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-credential-socket\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.133301 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133158 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-podinfo\") pod \"467d4202-b55f-4b78-8f8c-07184fbb79b1\" (UID: \"467d4202-b55f-4b78-8f8c-07184fbb79b1\") " Oct 31 01:29:41.133462 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133424 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-credential-socket" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:41.133882 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133770 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.133882 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133829 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.133882 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133844 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.133882 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.133858 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467d4202-b55f-4b78-8f8c-07184fbb79b1-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.134301 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.134268 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-certs" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:41.148848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.136160 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-envoy" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:41.148848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.136490 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-token" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:41.148848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.136601 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-podinfo" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:29:41.148848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.136891 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-kube-api-access-8rxt5" pod "467d4202-b55f-4b78-8f8c-07184fbb79b1" (UID: "467d4202-b55f-4b78-8f8c-07184fbb79b1"). InnerVolumeSpecName "kube-api-access-8rxt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:41.235614 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.235573 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxt5\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-kube-api-access-8rxt5\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.235614 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.235616 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.235911 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.235631 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.235911 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.235645 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.235911 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.235659 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467d4202-b55f-4b78-8f8c-07184fbb79b1-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.716627 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.716590 5890 generic.go:356] "Generic (PLEG): container finished" podID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerID="aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57" exitCode=0 Oct 31 01:29:41.717089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.716651 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" Oct 31 01:29:41.717089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.716683 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" event={"ID":"467d4202-b55f-4b78-8f8c-07184fbb79b1","Type":"ContainerDied","Data":"aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57"} Oct 31 01:29:41.717089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.716759 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp" event={"ID":"467d4202-b55f-4b78-8f8c-07184fbb79b1","Type":"ContainerDied","Data":"d5e2e1fcb9452512eb3f25dba1bc5ca131a2fd0fc903568e65bc22b85312b8ec"} Oct 31 01:29:41.717089 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.716813 5890 scope.go:122] "RemoveContainer" containerID="aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57" Oct 31 01:29:41.727755 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.727733 5890 scope.go:122] "RemoveContainer" containerID="b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616" Oct 31 01:29:41.739433 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.739410 5890 scope.go:122] "RemoveContainer" containerID="aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57" Oct 31 01:29:41.739727 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:41.739707 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57\": container with ID starting with aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57 not found: ID does not exist" containerID="aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57" Oct 31 01:29:41.739779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.739734 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57"} err="failed to get container status \"aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57\": rpc error: code = NotFound desc = could not find container \"aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57\": container with ID starting with aec057ebc64cc0c087d5abcdd9facbff1ff7b4712ab5c4d0aac61ee4ff542c57 not found: ID does not exist" Oct 31 01:29:41.739779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.739755 5890 scope.go:122] "RemoveContainer" containerID="b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616" Oct 31 01:29:41.740035 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:41.740014 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616\": container with ID starting with b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616 not found: ID does not exist" containerID="b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616" Oct 31 01:29:41.740139 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.740044 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616"} err="failed to get container status \"b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616\": rpc error: code = NotFound desc = could not find container \"b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616\": container with ID starting with b4e62eb754ed561a797f145f9f0c5a9d4c80bc9aa8797578fd9df75595b22616 not found: ID does not exist" Oct 31 01:29:41.877611 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.877396 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp"] Oct 31 01:29:41.923078 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.923038 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-bmpwp"] Oct 31 01:29:41.935966 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.935935 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" path="/var/lib/kubelet/pods/467d4202-b55f-4b78-8f8c-07184fbb79b1/volumes" Oct 31 01:29:41.998030 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:41.997944 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl"] Oct 31 01:29:42.725564 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.725485 5890 generic.go:356] "Generic (PLEG): container finished" podID="42be952b-15bf-41c7-b416-3f259d97849b" containerID="c027315a48fb1c90ab4423beb4d93f7fee0d632c1550f8b5b26470c6baae8659" exitCode=0 Oct 31 01:29:42.725999 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.725560 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" event={"ID":"42be952b-15bf-41c7-b416-3f259d97849b","Type":"ContainerDied","Data":"c027315a48fb1c90ab4423beb4d93f7fee0d632c1550f8b5b26470c6baae8659"} Oct 31 01:29:42.725999 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.725980 5890 scope.go:122] "RemoveContainer" containerID="c027315a48fb1c90ab4423beb4d93f7fee0d632c1550f8b5b26470c6baae8659" Oct 31 01:29:42.727426 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.727400 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:29:42.727496 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.727470 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:29:42.727533 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.727515 5890 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:29:42.849239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.849144 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw"] Oct 31 01:29:42.863803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:42.862849 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf"] Oct 31 01:29:43.733641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:43.733594 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" event={"ID":"42be952b-15bf-41c7-b416-3f259d97849b","Type":"ContainerStarted","Data":"2736fd97e93a33c6566e8b1a4629f01a728404b0f7883d03e0751a0590a6d95b"} Oct 31 01:29:44.093658 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:44.093575 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:29:44.093867 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:44.093842 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 01:29:44.093919 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:44.093898 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 01:29:45.093251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:45.093218 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 01:29:45.093653 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:45.093273 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 01:29:46.093889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.093852 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 01:29:46.094289 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.093911 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 01:29:46.531493 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.531458 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p"] Oct 31 01:29:46.531997 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.531983 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" Oct 31 01:29:46.532057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.531998 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" Oct 31 01:29:46.532057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532014 5890 cpu_manager.go:414] "containerMap: removing container" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" Oct 31 01:29:46.532057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532020 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" Oct 31 01:29:46.532057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532039 5890 cpu_manager.go:414] "containerMap: removing container" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" Oct 31 01:29:46.532057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532045 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" Oct 31 01:29:46.532057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532053 5890 cpu_manager.go:414] "containerMap: removing container" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" Oct 31 01:29:46.532057 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532058 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532067 5890 cpu_manager.go:414] "containerMap: removing container" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532072 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532088 5890 cpu_manager.go:414] "containerMap: removing container" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532093 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532100 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" containerName="maas-api" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532104 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" containerName="maas-api" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532116 5890 cpu_manager.go:414] "containerMap: removing container" podUID="be4c13ff-9583-4f2e-909e-549e1286ea06" containerName="maas-api" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532121 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4c13ff-9583-4f2e-909e-549e1286ea06" containerName="maas-api" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532190 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532199 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9096a8-7cf2-4454-a1c3-73cea55b63fc" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532208 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532215 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd07ff5-76c6-44e8-8c1a-7f76192bb5f8" containerName="maas-api" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532222 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4c13ff-9583-4f2e-909e-549e1286ea06" containerName="maas-api" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532229 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" Oct 31 01:29:46.532288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.532236 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d45daef-a5b8-4c2f-9b19-9e74a27288e2" containerName="istio-proxy" Oct 31 01:29:46.536047 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.536024 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:46.587151 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.587117 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6bd7c8\"" Oct 31 01:29:46.587316 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.587199 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6bd7c8-dockercfg-g77nb\"" Oct 31 01:29:46.594871 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.594845 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p"] Oct 31 01:29:46.674746 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.674709 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9"] Oct 31 01:29:46.675382 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.675366 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="467d4202-b55f-4b78-8f8c-07184fbb79b1" containerName="istio-proxy" Oct 31 01:29:46.678603 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.678588 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.707814 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.707770 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj82r\" (UniqueName: \"kubernetes.io/projected/8c4268ea-a56f-424c-b547-0bf580665e33-kube-api-access-hj82r\") pod \"payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:46.707969 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.707909 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4268ea-a56f-424c-b547-0bf580665e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:46.733140 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.733111 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9"] Oct 31 01:29:46.809171 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.809081 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4899c6c7-5566-4f4c-adb6-9e6c696d4838-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.809171 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.809132 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj82r\" (UniqueName: \"kubernetes.io/projected/8c4268ea-a56f-424c-b547-0bf580665e33-kube-api-access-hj82r\") pod \"payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:46.809424 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.809246 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4268ea-a56f-424c-b547-0bf580665e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:46.809424 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.809300 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kd6l\" (UniqueName: \"kubernetes.io/projected/4899c6c7-5566-4f4c-adb6-9e6c696d4838-kube-api-access-9kd6l\") pod \"payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.810200 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.810178 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4268ea-a56f-424c-b547-0bf580665e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:46.872069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.871916 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj82r\" (UniqueName: \"kubernetes.io/projected/8c4268ea-a56f-424c-b547-0bf580665e33-kube-api-access-hj82r\") pod \"payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:46.910423 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.910383 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4899c6c7-5566-4f4c-adb6-9e6c696d4838-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.910571 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.910456 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd6l\" (UniqueName: \"kubernetes.io/projected/4899c6c7-5566-4f4c-adb6-9e6c696d4838-kube-api-access-9kd6l\") pod \"payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.911356 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.911330 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4899c6c7-5566-4f4c-adb6-9e6c696d4838-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.976810 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.976472 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd6l\" (UniqueName: \"kubernetes.io/projected/4899c6c7-5566-4f4c-adb6-9e6c696d4838-kube-api-access-9kd6l\") pod \"payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.992097 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.992062 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:46.998401 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:46.998215 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" podUID="7d67e149-e4ad-489b-b42f-f53232be9311" containerName="payload-processing" containerID="cri-o://5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df" gracePeriod=25 Oct 31 01:29:47.094020 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.093985 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 01:29:47.094463 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.094043 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 01:29:47.169079 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.169043 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:47.394421 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.394386 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9"] Oct 31 01:29:47.395109 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:29:47.395076 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4899c6c7_5566_4f4c_adb6_9e6c696d4838.slice/crio-8ad05e06f10036dc952af1585f68592e6c30200e34515c515f94c8fcfd88a114 WatchSource:0}: Error finding container 8ad05e06f10036dc952af1585f68592e6c30200e34515c515f94c8fcfd88a114: Status 404 returned error can't find the container with id 8ad05e06f10036dc952af1585f68592e6c30200e34515c515f94c8fcfd88a114 Oct 31 01:29:47.542015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.541989 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p"] Oct 31 01:29:47.543774 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:29:47.543743 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4268ea_a56f_424c_b547_0bf580665e33.slice/crio-34bae8353854ce909708be071e28f0d0695d361b13ffc26f59e9e60ddcb356b4 WatchSource:0}: Error finding container 34bae8353854ce909708be071e28f0d0695d361b13ffc26f59e9e60ddcb356b4: Status 404 returned error can't find the container with id 34bae8353854ce909708be071e28f0d0695d361b13ffc26f59e9e60ddcb356b4 Oct 31 01:29:47.791139 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.791089 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" event={"ID":"4899c6c7-5566-4f4c-adb6-9e6c696d4838","Type":"ContainerStarted","Data":"68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb"} Oct 31 01:29:47.791357 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.791171 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:47.791357 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.791195 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" event={"ID":"4899c6c7-5566-4f4c-adb6-9e6c696d4838","Type":"ContainerStarted","Data":"8ad05e06f10036dc952af1585f68592e6c30200e34515c515f94c8fcfd88a114"} Oct 31 01:29:47.793384 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.793351 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" event={"ID":"8c4268ea-a56f-424c-b547-0bf580665e33","Type":"ContainerStarted","Data":"58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd"} Oct 31 01:29:47.793516 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.793403 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" event={"ID":"8c4268ea-a56f-424c-b547-0bf580665e33","Type":"ContainerStarted","Data":"34bae8353854ce909708be071e28f0d0695d361b13ffc26f59e9e60ddcb356b4"} Oct 31 01:29:47.793516 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.793483 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:47.902573 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:47.902525 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" podStartSLOduration=1.9025125429999998 podStartE2EDuration="1.902512543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:47.901954925 +0000 UTC m=+3680.104205359" watchObservedRunningTime="2026-10-31 01:29:47.902512543 +0000 UTC m=+3680.104762972" Oct 31 01:29:48.054039 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:48.053904 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" podStartSLOduration=2.053884625 podStartE2EDuration="2.053884625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:48.043582308 +0000 UTC m=+3680.245832742" watchObservedRunningTime="2026-10-31 01:29:48.053884625 +0000 UTC m=+3680.256135059" Oct 31 01:29:48.093717 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:48.093669 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 01:29:48.093903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:48.093733 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 01:29:49.093999 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:49.093960 5890 patch_prober.go:29] interesting pod/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 01:29:49.094587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:49.094017 5890 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 01:29:50.000937 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.000896 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" containerID="cri-o://a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c" gracePeriod=45 Oct 31 01:29:50.004067 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.004032 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" podUID="4e45c970-b6d8-48e0-b780-ec5444252491" containerName="main" containerID="cri-o://2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7" gracePeriod=45 Oct 31 01:29:50.096889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.096859 5890 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:29:50.097352 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.097332 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:29:50.098094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.098076 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:29:50.135386 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.135350 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": dial tcp 10.132.0.99:8000: connect: connection refused" Oct 31 01:29:50.159340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.159291 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" podUID="4e45c970-b6d8-48e0-b780-ec5444252491" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": dial tcp 10.132.0.102:8000: connect: connection refused" Oct 31 01:29:50.267397 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.267368 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 01:29:50.299173 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.299146 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 01:29:50.348481 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348452 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-model-cache\") pod \"d04083cb-7d47-4d40-8b31-76005517baa1\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " Oct 31 01:29:50.348656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348517 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-home\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-home\") pod \"d04083cb-7d47-4d40-8b31-76005517baa1\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " Oct 31 01:29:50.348656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348542 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-model-cache\") pod \"4e45c970-b6d8-48e0-b780-ec5444252491\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " Oct 31 01:29:50.348656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348580 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-home\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-home\") pod \"4e45c970-b6d8-48e0-b780-ec5444252491\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " Oct 31 01:29:50.348656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348604 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-tmp-dir\") pod \"d04083cb-7d47-4d40-8b31-76005517baa1\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " Oct 31 01:29:50.348914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348704 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04083cb-7d47-4d40-8b31-76005517baa1-kube-api-access-4jr7t\" (UniqueName: \"kubernetes.io/projected/d04083cb-7d47-4d40-8b31-76005517baa1-kube-api-access-4jr7t\") pod \"d04083cb-7d47-4d40-8b31-76005517baa1\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " Oct 31 01:29:50.348914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348734 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e45c970-b6d8-48e0-b780-ec5444252491-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e45c970-b6d8-48e0-b780-ec5444252491-tls-certs\") pod \"4e45c970-b6d8-48e0-b780-ec5444252491\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " Oct 31 01:29:50.348914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348752 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-home" pod "d04083cb-7d47-4d40-8b31-76005517baa1" (UID: "d04083cb-7d47-4d40-8b31-76005517baa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.348914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348771 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-model-cache" pod "d04083cb-7d47-4d40-8b31-76005517baa1" (UID: "d04083cb-7d47-4d40-8b31-76005517baa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.348914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348811 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d04083cb-7d47-4d40-8b31-76005517baa1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d04083cb-7d47-4d40-8b31-76005517baa1-tls-certs\") pod \"d04083cb-7d47-4d40-8b31-76005517baa1\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " Oct 31 01:29:50.348914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348839 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-dshm\") pod \"d04083cb-7d47-4d40-8b31-76005517baa1\" (UID: \"d04083cb-7d47-4d40-8b31-76005517baa1\") " Oct 31 01:29:50.348914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348900 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-model-cache" pod "4e45c970-b6d8-48e0-b780-ec5444252491" (UID: "4e45c970-b6d8-48e0-b780-ec5444252491"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.349243 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348923 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-home" pod "4e45c970-b6d8-48e0-b780-ec5444252491" (UID: "4e45c970-b6d8-48e0-b780-ec5444252491"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.349243 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.348997 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-tmp-dir" pod "d04083cb-7d47-4d40-8b31-76005517baa1" (UID: "d04083cb-7d47-4d40-8b31-76005517baa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.349328 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.349308 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.349377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.349329 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.349377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.349345 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.349377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.349358 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.349377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.349369 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.351100 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.351075 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-dshm" pod "d04083cb-7d47-4d40-8b31-76005517baa1" (UID: "d04083cb-7d47-4d40-8b31-76005517baa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.351182 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.351123 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e45c970-b6d8-48e0-b780-ec5444252491-tls-certs" pod "4e45c970-b6d8-48e0-b780-ec5444252491" (UID: "4e45c970-b6d8-48e0-b780-ec5444252491"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:50.351373 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.351348 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04083cb-7d47-4d40-8b31-76005517baa1-tls-certs" pod "d04083cb-7d47-4d40-8b31-76005517baa1" (UID: "d04083cb-7d47-4d40-8b31-76005517baa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:50.351472 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.351378 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04083cb-7d47-4d40-8b31-76005517baa1-kube-api-access-4jr7t" pod "d04083cb-7d47-4d40-8b31-76005517baa1" (UID: "d04083cb-7d47-4d40-8b31-76005517baa1"). InnerVolumeSpecName "kube-api-access-4jr7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:50.450244 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450214 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e45c970-b6d8-48e0-b780-ec5444252491-kube-api-access-7tn8h\" (UniqueName: \"kubernetes.io/projected/4e45c970-b6d8-48e0-b780-ec5444252491-kube-api-access-7tn8h\") pod \"4e45c970-b6d8-48e0-b780-ec5444252491\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " Oct 31 01:29:50.450413 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450333 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-tmp-dir\") pod \"4e45c970-b6d8-48e0-b780-ec5444252491\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " Oct 31 01:29:50.450413 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450379 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-dshm\") pod \"4e45c970-b6d8-48e0-b780-ec5444252491\" (UID: \"4e45c970-b6d8-48e0-b780-ec5444252491\") " Oct 31 01:29:50.450621 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450596 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-tmp-dir" pod "4e45c970-b6d8-48e0-b780-ec5444252491" (UID: "4e45c970-b6d8-48e0-b780-ec5444252491"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.450768 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450753 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jr7t\" (UniqueName: \"kubernetes.io/projected/d04083cb-7d47-4d40-8b31-76005517baa1-kube-api-access-4jr7t\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.450850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450771 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e45c970-b6d8-48e0-b780-ec5444252491-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.450850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450807 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04083cb-7d47-4d40-8b31-76005517baa1-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.450850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450820 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04083cb-7d47-4d40-8b31-76005517baa1-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.450850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.450830 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.452419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.452395 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-dshm" pod "4e45c970-b6d8-48e0-b780-ec5444252491" (UID: "4e45c970-b6d8-48e0-b780-ec5444252491"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.452626 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.452597 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e45c970-b6d8-48e0-b780-ec5444252491-kube-api-access-7tn8h" pod "4e45c970-b6d8-48e0-b780-ec5444252491" (UID: "4e45c970-b6d8-48e0-b780-ec5444252491"). InnerVolumeSpecName "kube-api-access-7tn8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:50.552838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.552720 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e45c970-b6d8-48e0-b780-ec5444252491-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.552838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.552754 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tn8h\" (UniqueName: \"kubernetes.io/projected/4e45c970-b6d8-48e0-b780-ec5444252491-kube-api-access-7tn8h\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.814766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.814678 5890 generic.go:356] "Generic (PLEG): container finished" podID="4e45c970-b6d8-48e0-b780-ec5444252491" containerID="2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7" exitCode=0 Oct 31 01:29:50.814766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.814719 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" Oct 31 01:29:50.814766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.814757 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" event={"ID":"4e45c970-b6d8-48e0-b780-ec5444252491","Type":"ContainerDied","Data":"2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7"} Oct 31 01:29:50.815070 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.814827 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6" event={"ID":"4e45c970-b6d8-48e0-b780-ec5444252491","Type":"ContainerDied","Data":"ac4ea998480539596a8dd9e414335ce62ec3a8d65d4be76e55d60f9a4e71f687"} Oct 31 01:29:50.815070 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.814853 5890 scope.go:122] "RemoveContainer" containerID="2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7" Oct 31 01:29:50.816232 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.816206 5890 generic.go:356] "Generic (PLEG): container finished" podID="d04083cb-7d47-4d40-8b31-76005517baa1" containerID="a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c" exitCode=0 Oct 31 01:29:50.816355 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.816246 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" Oct 31 01:29:50.816355 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.816288 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" event={"ID":"d04083cb-7d47-4d40-8b31-76005517baa1","Type":"ContainerDied","Data":"a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c"} Oct 31 01:29:50.816355 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.816322 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz" event={"ID":"d04083cb-7d47-4d40-8b31-76005517baa1","Type":"ContainerDied","Data":"34a9248783da084bd354df807656b1113a554f668d6c39c5e1013d79835d9ca8"} Oct 31 01:29:50.826641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.826619 5890 scope.go:122] "RemoveContainer" containerID="2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7" Oct 31 01:29:50.827064 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:50.826943 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7\": container with ID starting with 2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7 not found: ID does not exist" containerID="2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7" Oct 31 01:29:50.827064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.826978 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7"} err="failed to get container status \"2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7\": rpc error: code = NotFound desc = could not find container \"2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7\": container with ID starting with 2e96c600194da13efb91097059679ba1d9679f009bbff23ebf932bf40a2eb3e7 not found: ID does not exist" Oct 31 01:29:50.827064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.827002 5890 scope.go:122] "RemoveContainer" containerID="a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c" Oct 31 01:29:50.835660 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.835645 5890 scope.go:122] "RemoveContainer" containerID="a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c" Oct 31 01:29:50.835965 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:50.835945 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c\": container with ID starting with a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c not found: ID does not exist" containerID="a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c" Oct 31 01:29:50.836039 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.835975 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c"} err="failed to get container status \"a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c\": rpc error: code = NotFound desc = could not find container \"a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c\": container with ID starting with a983e8a81d690a163ac24073dacf3ff215e266790729575a75604f7f42455c2c not found: ID does not exist" Oct 31 01:29:50.938850 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.938819 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf"] Oct 31 01:29:50.971979 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.971953 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6"] Oct 31 01:29:50.977092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:50.977066 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-2-w1-b63c24-kserve-77548c5479-b2jq6"] Oct 31 01:29:51.049079 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:51.049047 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f"] Oct 31 01:29:51.081004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:51.080929 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz"] Oct 31 01:29:51.085072 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:51.085045 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/distinct-w1-b63c24-kserve-5f5d5bb7d8-vxdkz"] Oct 31 01:29:51.125322 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:51.125283 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" podUID="7d67e149-e4ad-489b-b42f-f53232be9311" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:29:51.933167 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:51.933137 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e45c970-b6d8-48e0-b780-ec5444252491" path="/var/lib/kubelet/pods/4e45c970-b6d8-48e0-b780-ec5444252491/volumes" Oct 31 01:29:51.933564 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:51.933551 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" path="/var/lib/kubelet/pods/d04083cb-7d47-4d40-8b31-76005517baa1/volumes" Oct 31 01:29:55.939534 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:55.939497 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" podUID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" containerName="payload-processing" containerID="cri-o://5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1" gracePeriod=25 Oct 31 01:29:56.050014 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:56.049964 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" podUID="71d79d95-9651-4aed-a942-e4d087b335d5" containerName="payload-pre-processing" containerID="cri-o://c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b" gracePeriod=25 Oct 31 01:29:57.903915 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:57.903872 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" containerID="cri-o://fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc" gracePeriod=45 Oct 31 01:29:57.915241 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:57.915199 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" containerID="cri-o://bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85" gracePeriod=45 Oct 31 01:29:58.074425 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.074110 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": dial tcp 10.132.0.96:8000: connect: connection refused" Oct 31 01:29:58.194864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.194779 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 01:29:58.214100 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.214070 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 01:29:58.351873 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.351842 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-tmp-dir\") pod \"7b742d5e-5806-4447-b956-c9f443007ea5\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " Oct 31 01:29:58.352071 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.351903 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b742d5e-5806-4447-b956-c9f443007ea5-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b742d5e-5806-4447-b956-c9f443007ea5-tls-certs\") pod \"7b742d5e-5806-4447-b956-c9f443007ea5\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " Oct 31 01:29:58.352071 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.351963 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-dshm\") pod \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " Oct 31 01:29:58.352071 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352020 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-dshm\") pod \"7b742d5e-5806-4447-b956-c9f443007ea5\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " Oct 31 01:29:58.352071 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352044 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-home\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-home\") pod \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " Oct 31 01:29:58.352296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352117 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tmp-dir\") pod \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " Oct 31 01:29:58.352296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352115 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-tmp-dir" pod "7b742d5e-5806-4447-b956-c9f443007ea5" (UID: "7b742d5e-5806-4447-b956-c9f443007ea5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.352296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352160 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-kube-api-access-fzc7w\" (UniqueName: \"kubernetes.io/projected/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-kube-api-access-fzc7w\") pod \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " Oct 31 01:29:58.352296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352202 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-home\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-home\") pod \"7b742d5e-5806-4447-b956-c9f443007ea5\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " Oct 31 01:29:58.352296 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352236 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-model-cache\") pod \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " Oct 31 01:29:58.352559 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352300 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b742d5e-5806-4447-b956-c9f443007ea5-kube-api-access-frrqs\" (UniqueName: \"kubernetes.io/projected/7b742d5e-5806-4447-b956-c9f443007ea5-kube-api-access-frrqs\") pod \"7b742d5e-5806-4447-b956-c9f443007ea5\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " Oct 31 01:29:58.352559 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352339 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tls-certs\") pod \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\" (UID: \"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60\") " Oct 31 01:29:58.352559 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352369 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-model-cache\") pod \"7b742d5e-5806-4447-b956-c9f443007ea5\" (UID: \"7b742d5e-5806-4447-b956-c9f443007ea5\") " Oct 31 01:29:58.352559 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352396 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tmp-dir" pod "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" (UID: "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.352889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352553 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-home" pod "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" (UID: "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.352889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352613 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-model-cache" pod "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" (UID: "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.352889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352742 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-home" pod "7b742d5e-5806-4447-b956-c9f443007ea5" (UID: "7b742d5e-5806-4447-b956-c9f443007ea5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.353093 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.352963 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-model-cache" pod "7b742d5e-5806-4447-b956-c9f443007ea5" (UID: "7b742d5e-5806-4447-b956-c9f443007ea5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.353189 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.353168 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353255 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.353196 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353255 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.353213 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353255 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.353226 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353255 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.353240 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.354766 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.354736 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b742d5e-5806-4447-b956-c9f443007ea5-tls-certs" pod "7b742d5e-5806-4447-b956-c9f443007ea5" (UID: "7b742d5e-5806-4447-b956-c9f443007ea5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:58.355338 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.355305 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-dshm" pod "7b742d5e-5806-4447-b956-c9f443007ea5" (UID: "7b742d5e-5806-4447-b956-c9f443007ea5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.355593 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.355570 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b742d5e-5806-4447-b956-c9f443007ea5-kube-api-access-frrqs" pod "7b742d5e-5806-4447-b956-c9f443007ea5" (UID: "7b742d5e-5806-4447-b956-c9f443007ea5"). InnerVolumeSpecName "kube-api-access-frrqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:58.355668 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.355615 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tls-certs" pod "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" (UID: "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:58.355841 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.355765 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-dshm" pod "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" (UID: "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.355980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.355961 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-kube-api-access-fzc7w" pod "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" (UID: "0f49a78e-3845-4cf1-97d0-1d6ea81b3d60"). InnerVolumeSpecName "kube-api-access-fzc7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:58.454636 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.454596 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzc7w\" (UniqueName: \"kubernetes.io/projected/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-kube-api-access-fzc7w\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.454636 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.454637 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrqs\" (UniqueName: \"kubernetes.io/projected/7b742d5e-5806-4447-b956-c9f443007ea5-kube-api-access-frrqs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.455069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.454654 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.455069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.454670 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.455069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.454714 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b742d5e-5806-4447-b956-c9f443007ea5-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.455069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.454729 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.455069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.454742 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b742d5e-5806-4447-b956-c9f443007ea5-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.803652 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.803578 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:29:58.803652 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.803647 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:29:58.872198 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.872167 5890 generic.go:356] "Generic (PLEG): container finished" podID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerID="fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc" exitCode=0 Oct 31 01:29:58.872469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.872241 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" Oct 31 01:29:58.872469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.872261 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" event={"ID":"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60","Type":"ContainerDied","Data":"fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc"} Oct 31 01:29:58.872469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.872308 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw" event={"ID":"0f49a78e-3845-4cf1-97d0-1d6ea81b3d60","Type":"ContainerDied","Data":"5e6caefe9ccc10ef3e17a52edcd87f9c49e3adc937af6ec70c52c8029d3b7274"} Oct 31 01:29:58.872469 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.872326 5890 scope.go:122] "RemoveContainer" containerID="fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc" Oct 31 01:29:58.873879 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.873849 5890 generic.go:356] "Generic (PLEG): container finished" podID="7b742d5e-5806-4447-b956-c9f443007ea5" containerID="bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85" exitCode=0 Oct 31 01:29:58.873985 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.873903 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" event={"ID":"7b742d5e-5806-4447-b956-c9f443007ea5","Type":"ContainerDied","Data":"bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85"} Oct 31 01:29:58.873985 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.873932 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" event={"ID":"7b742d5e-5806-4447-b956-c9f443007ea5","Type":"ContainerDied","Data":"a37392929bdc65afb2dffa57ee22ac27669a14978e7ad1fc4edc12248a3b3456"} Oct 31 01:29:58.874073 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.874005 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf" Oct 31 01:29:58.886285 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.886265 5890 scope.go:122] "RemoveContainer" containerID="fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc" Oct 31 01:29:58.887145 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:58.887035 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc\": container with ID starting with fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc not found: ID does not exist" containerID="fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc" Oct 31 01:29:58.887145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.887085 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc"} err="failed to get container status \"fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc\": rpc error: code = NotFound desc = could not find container \"fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc\": container with ID starting with fcc28533beae99000afe3672f1a3b970befbc8eff77d8fe623afefc59f1c4cbc not found: ID does not exist" Oct 31 01:29:58.887145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.887111 5890 scope.go:122] "RemoveContainer" containerID="bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85" Oct 31 01:29:58.898055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.898030 5890 scope.go:122] "RemoveContainer" containerID="bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85" Oct 31 01:29:58.898458 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:29:58.898434 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85\": container with ID starting with bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85 not found: ID does not exist" containerID="bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85" Oct 31 01:29:58.898532 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:58.898466 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85"} err="failed to get container status \"bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85\": rpc error: code = NotFound desc = could not find container \"bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85\": container with ID starting with bdd497e5cf5a3708cd047ceb778c5fd65c129b82f8a178abb8937bf644b1cf85 not found: ID does not exist" Oct 31 01:29:59.136813 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:59.136586 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf"] Oct 31 01:29:59.141144 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:59.141110 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-w2-29ece5-kserve-8545b8dcb4-cdlkf"] Oct 31 01:29:59.246274 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:59.246239 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw"] Oct 31 01:29:59.249758 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:59.249731 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/distinct-2-w2-29ece5-kserve-f4bfb9895-cpvzw"] Oct 31 01:29:59.933288 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:59.933255 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" path="/var/lib/kubelet/pods/0f49a78e-3845-4cf1-97d0-1d6ea81b3d60/volumes" Oct 31 01:29:59.933874 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:29:59.933857 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" path="/var/lib/kubelet/pods/7b742d5e-5806-4447-b956-c9f443007ea5/volumes" Oct 31 01:30:00.199533 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.199443 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf"] Oct 31 01:30:00.200191 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200168 5890 cpu_manager.go:414] "containerMap: removing container" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" Oct 31 01:30:00.200305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200192 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" Oct 31 01:30:00.200305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200242 5890 cpu_manager.go:414] "containerMap: removing container" podUID="4e45c970-b6d8-48e0-b780-ec5444252491" containerName="main" Oct 31 01:30:00.200305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200251 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e45c970-b6d8-48e0-b780-ec5444252491" containerName="main" Oct 31 01:30:00.200305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200294 5890 cpu_manager.go:414] "containerMap: removing container" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" Oct 31 01:30:00.200305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200302 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" Oct 31 01:30:00.200567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200314 5890 cpu_manager.go:414] "containerMap: removing container" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" Oct 31 01:30:00.200567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200321 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" Oct 31 01:30:00.200567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200448 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e45c970-b6d8-48e0-b780-ec5444252491" containerName="main" Oct 31 01:30:00.200567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200468 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04083cb-7d47-4d40-8b31-76005517baa1" containerName="main" Oct 31 01:30:00.200567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200479 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b742d5e-5806-4447-b956-c9f443007ea5" containerName="main" Oct 31 01:30:00.200567 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.200490 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f49a78e-3845-4cf1-97d0-1d6ea81b3d60" containerName="main" Oct 31 01:30:00.205899 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.205874 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.237346 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.237320 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd"] Oct 31 01:30:00.241751 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.241730 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.265948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.265923 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:30:00.298767 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.298725 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf"] Oct 31 01:30:00.359666 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.359630 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd"] Oct 31 01:30:00.375604 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.375572 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e078e70d-eeea-437a-994a-0a8a0c68f831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.375817 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.375633 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.375817 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.375697 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8z6k\" (UniqueName: \"kubernetes.io/projected/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-kube-api-access-b8z6k\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.375817 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.375725 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thrj6\" (UniqueName: \"kubernetes.io/projected/e078e70d-eeea-437a-994a-0a8a0c68f831-kube-api-access-thrj6\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.476957 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.476854 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e078e70d-eeea-437a-994a-0a8a0c68f831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.476957 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.476946 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.477192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.476983 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8z6k\" (UniqueName: \"kubernetes.io/projected/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-kube-api-access-b8z6k\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.477192 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.477026 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thrj6\" (UniqueName: \"kubernetes.io/projected/e078e70d-eeea-437a-994a-0a8a0c68f831-kube-api-access-thrj6\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.477613 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.477589 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e078e70d-eeea-437a-994a-0a8a0c68f831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.477727 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.477683 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.542622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.542592 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8z6k\" (UniqueName: \"kubernetes.io/projected/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-kube-api-access-b8z6k\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.548181 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.548154 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thrj6\" (UniqueName: \"kubernetes.io/projected/e078e70d-eeea-437a-994a-0a8a0c68f831-kube-api-access-thrj6\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.555446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.555426 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:00.818864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.818797 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:00.924845 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.924812 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd"] Oct 31 01:30:00.926326 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:30:00.926289 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5ceaed_5e34_4a0e_bf28_429fb7bfbce2.slice/crio-15680b1992824da83c437983898d1bd870ecef90d4bbb3e07ecfd7c7644dc0b8 WatchSource:0}: Error finding container 15680b1992824da83c437983898d1bd870ecef90d4bbb3e07ecfd7c7644dc0b8: Status 404 returned error can't find the container with id 15680b1992824da83c437983898d1bd870ecef90d4bbb3e07ecfd7c7644dc0b8 Oct 31 01:30:00.967441 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.967409 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" podUID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:00.967555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:00.967448 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" podUID="71d79d95-9651-4aed-a942-e4d087b335d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:01.126381 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:01.126263 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" podUID="7d67e149-e4ad-489b-b42f-f53232be9311" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:01.188798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:01.188753 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf"] Oct 31 01:30:01.190265 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:30:01.190231 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode078e70d_eeea_437a_994a_0a8a0c68f831.slice/crio-3ad170cd21bfe3a41ea5e3bcaa2be8b29edac5de44859385a92dced09eaa81d4 WatchSource:0}: Error finding container 3ad170cd21bfe3a41ea5e3bcaa2be8b29edac5de44859385a92dced09eaa81d4: Status 404 returned error can't find the container with id 3ad170cd21bfe3a41ea5e3bcaa2be8b29edac5de44859385a92dced09eaa81d4 Oct 31 01:30:01.900655 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:01.900549 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" event={"ID":"e078e70d-eeea-437a-994a-0a8a0c68f831","Type":"ContainerStarted","Data":"1b33a58b64a7efa0cc0e12bdb95782bd48431c2ab3e130781d5743dcde98251d"} Oct 31 01:30:01.900655 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:01.900648 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" event={"ID":"e078e70d-eeea-437a-994a-0a8a0c68f831","Type":"ContainerStarted","Data":"3ad170cd21bfe3a41ea5e3bcaa2be8b29edac5de44859385a92dced09eaa81d4"} Oct 31 01:30:01.902147 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:01.902117 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" event={"ID":"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2","Type":"ContainerStarted","Data":"f765de2f1aad1b232caa8da3a36e5f852ddc4131682200e210dc5abf19738844"} Oct 31 01:30:01.902251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:01.902163 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" event={"ID":"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2","Type":"ContainerStarted","Data":"15680b1992824da83c437983898d1bd870ecef90d4bbb3e07ecfd7c7644dc0b8"} Oct 31 01:30:02.012945 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.012892 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" podStartSLOduration=2.012877149 podStartE2EDuration="2.012877149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.011223239 +0000 UTC m=+3694.213473672" watchObservedRunningTime="2026-10-31 01:30:02.012877149 +0000 UTC m=+3694.215127581" Oct 31 01:30:02.123933 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.123876 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" podStartSLOduration=2.123863812 podStartE2EDuration="2.123863812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.121232393 +0000 UTC m=+3694.323482827" watchObservedRunningTime="2026-10-31 01:30:02.123863812 +0000 UTC m=+3694.326114245" Oct 31 01:30:02.909983 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.909950 5890 generic.go:356] "Generic (PLEG): container finished" podID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerID="f765de2f1aad1b232caa8da3a36e5f852ddc4131682200e210dc5abf19738844" exitCode=60 Oct 31 01:30:02.910390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.910037 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" event={"ID":"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2","Type":"ContainerDied","Data":"f765de2f1aad1b232caa8da3a36e5f852ddc4131682200e210dc5abf19738844"} Oct 31 01:30:02.910390 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.910360 5890 scope.go:122] "RemoveContainer" containerID="f765de2f1aad1b232caa8da3a36e5f852ddc4131682200e210dc5abf19738844" Oct 31 01:30:02.912109 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.912082 5890 generic.go:356] "Generic (PLEG): container finished" podID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerID="1b33a58b64a7efa0cc0e12bdb95782bd48431c2ab3e130781d5743dcde98251d" exitCode=60 Oct 31 01:30:02.912210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.912139 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" event={"ID":"e078e70d-eeea-437a-994a-0a8a0c68f831","Type":"ContainerDied","Data":"1b33a58b64a7efa0cc0e12bdb95782bd48431c2ab3e130781d5743dcde98251d"} Oct 31 01:30:02.912372 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:02.912360 5890 scope.go:122] "RemoveContainer" containerID="1b33a58b64a7efa0cc0e12bdb95782bd48431c2ab3e130781d5743dcde98251d" Oct 31 01:30:03.921730 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:03.921697 5890 generic.go:356] "Generic (PLEG): container finished" podID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerID="55f44378c2d086c8b5285727da56b2520ae27d7c446bf0b38027f272ec6db8a1" exitCode=60 Oct 31 01:30:03.922214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:03.921795 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" event={"ID":"e078e70d-eeea-437a-994a-0a8a0c68f831","Type":"ContainerDied","Data":"55f44378c2d086c8b5285727da56b2520ae27d7c446bf0b38027f272ec6db8a1"} Oct 31 01:30:03.922214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:03.921859 5890 scope.go:122] "RemoveContainer" containerID="1b33a58b64a7efa0cc0e12bdb95782bd48431c2ab3e130781d5743dcde98251d" Oct 31 01:30:03.922214 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:03.922134 5890 scope.go:122] "RemoveContainer" containerID="55f44378c2d086c8b5285727da56b2520ae27d7c446bf0b38027f272ec6db8a1" Oct 31 01:30:03.922465 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:03.922438 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf_odh-ai-gateway-infra(e078e70d-eeea-437a-994a-0a8a0c68f831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" Oct 31 01:30:03.923917 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:03.923895 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" event={"ID":"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2","Type":"ContainerStarted","Data":"99f34ec51bf7e24682b0876705d0206b4f0dce3247f56bfa94d8077203b1e199"} Oct 31 01:30:04.932404 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:04.932375 5890 scope.go:122] "RemoveContainer" containerID="55f44378c2d086c8b5285727da56b2520ae27d7c446bf0b38027f272ec6db8a1" Oct 31 01:30:04.932856 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:04.932631 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf_odh-ai-gateway-infra(e078e70d-eeea-437a-994a-0a8a0c68f831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" Oct 31 01:30:04.934018 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:04.933989 5890 generic.go:356] "Generic (PLEG): container finished" podID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerID="99f34ec51bf7e24682b0876705d0206b4f0dce3247f56bfa94d8077203b1e199" exitCode=60 Oct 31 01:30:04.934136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:04.934068 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" event={"ID":"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2","Type":"ContainerDied","Data":"99f34ec51bf7e24682b0876705d0206b4f0dce3247f56bfa94d8077203b1e199"} Oct 31 01:30:04.934136 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:04.934125 5890 scope.go:122] "RemoveContainer" containerID="f765de2f1aad1b232caa8da3a36e5f852ddc4131682200e210dc5abf19738844" Oct 31 01:30:04.934356 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:04.934340 5890 scope.go:122] "RemoveContainer" containerID="99f34ec51bf7e24682b0876705d0206b4f0dce3247f56bfa94d8077203b1e199" Oct 31 01:30:04.934583 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:04.934561 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd_odh-ai-gateway-infra(cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" Oct 31 01:30:05.394599 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.394505 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9"] Oct 31 01:30:05.399802 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.399764 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.432641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.432607 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcjn\" (UniqueName: \"kubernetes.io/projected/99739e84-48b2-485e-9cc3-4eca5e4fb179-kube-api-access-kwcjn\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.432815 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.432664 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.432815 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.432800 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.449950 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.449915 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:05.457174 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.457147 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9"] Oct 31 01:30:05.533889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.533846 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcjn\" (UniqueName: \"kubernetes.io/projected/99739e84-48b2-485e-9cc3-4eca5e4fb179-kube-api-access-kwcjn\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.534055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.533920 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.534055 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.533988 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.534605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.534581 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.536368 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.536346 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.592748 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.592714 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcjn\" (UniqueName: \"kubernetes.io/projected/99739e84-48b2-485e-9cc3-4eca5e4fb179-kube-api-access-kwcjn\") pod \"maas-api-revoke-sub-d2cfd12b-jsbb9\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.672460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.672427 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv"] Oct 31 01:30:05.677186 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.677161 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.713329 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.713302 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:05.735941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.735907 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv"] Oct 31 01:30:05.736138 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.736108 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfdv\" (UniqueName: \"kubernetes.io/projected/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-kube-api-access-hkfdv\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.736378 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.736351 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.736517 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.736407 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.837904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.837873 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.837904 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.837915 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.838126 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.837951 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfdv\" (UniqueName: \"kubernetes.io/projected/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-kube-api-access-hkfdv\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.838754 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.838723 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.840254 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.840226 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.896617 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.896593 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfdv\" (UniqueName: \"kubernetes.io/projected/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-kube-api-access-hkfdv\") pod \"maas-api-revoke-sub-776a4eb3-wxrfv\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:05.941439 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.941367 5890 scope.go:122] "RemoveContainer" containerID="99f34ec51bf7e24682b0876705d0206b4f0dce3247f56bfa94d8077203b1e199" Oct 31 01:30:05.941865 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:05.941650 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd_odh-ai-gateway-infra(cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" Oct 31 01:30:05.991575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:05.991548 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:06.097843 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:06.097800 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9"] Oct 31 01:30:06.133250 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:30:06.133220 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99739e84_48b2_485e_9cc3_4eca5e4fb179.slice/crio-583c2e8d7231c35b383db78b80cc48ff94816c2d393dafe43804d2eef97b759e WatchSource:0}: Error finding container 583c2e8d7231c35b383db78b80cc48ff94816c2d393dafe43804d2eef97b759e: Status 404 returned error can't find the container with id 583c2e8d7231c35b383db78b80cc48ff94816c2d393dafe43804d2eef97b759e Oct 31 01:30:06.361106 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:06.361076 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv"] Oct 31 01:30:06.363041 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:30:06.363010 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5e5a70_e6ab_4b55_836c_23a5e0a04fc1.slice/crio-779693db9198aa4cb780ec5dcd45134fb15b9475ad5b19e347a962b9947afafc WatchSource:0}: Error finding container 779693db9198aa4cb780ec5dcd45134fb15b9475ad5b19e347a962b9947afafc: Status 404 returned error can't find the container with id 779693db9198aa4cb780ec5dcd45134fb15b9475ad5b19e347a962b9947afafc Oct 31 01:30:06.950305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:06.950272 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" event={"ID":"99739e84-48b2-485e-9cc3-4eca5e4fb179","Type":"ContainerStarted","Data":"e8d87c70bd886f4cb64c2749df34783969a0d32f254b0ac6f7d8fa1d7c0bcfbf"} Oct 31 01:30:06.950811 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:06.950328 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" event={"ID":"99739e84-48b2-485e-9cc3-4eca5e4fb179","Type":"ContainerStarted","Data":"583c2e8d7231c35b383db78b80cc48ff94816c2d393dafe43804d2eef97b759e"} Oct 31 01:30:06.952042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:06.952013 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" event={"ID":"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1","Type":"ContainerStarted","Data":"3008249e0d68bb6e0bd19f88355a7b0fc12d2213e37e81209af3f9513755bde3"} Oct 31 01:30:06.952150 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:06.952054 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" event={"ID":"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1","Type":"ContainerStarted","Data":"779693db9198aa4cb780ec5dcd45134fb15b9475ad5b19e347a962b9947afafc"} Oct 31 01:30:07.020041 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:07.020006 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:30:07.063549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.063497 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" podStartSLOduration=2.063482672 podStartE2EDuration="2.063482672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:07.061849236 +0000 UTC m=+3699.264099662" watchObservedRunningTime="2026-10-31 01:30:07.063482672 +0000 UTC m=+3699.265733104" Oct 31 01:30:07.174674 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.174625 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" podStartSLOduration=2.174606713 podStartE2EDuration="2.174606713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:07.173631548 +0000 UTC m=+3699.375881982" watchObservedRunningTime="2026-10-31 01:30:07.174606713 +0000 UTC m=+3699.376857181" Oct 31 01:30:07.284362 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.284337 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 01:30:07.353970 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.353938 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d67e149-e4ad-489b-b42f-f53232be9311-kube-api-access-ncpm9\" (UniqueName: \"kubernetes.io/projected/7d67e149-e4ad-489b-b42f-f53232be9311-kube-api-access-ncpm9\") pod \"7d67e149-e4ad-489b-b42f-f53232be9311\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " Oct 31 01:30:07.354115 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.354045 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d67e149-e4ad-489b-b42f-f53232be9311-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d67e149-e4ad-489b-b42f-f53232be9311-plugins-config-volume\") pod \"7d67e149-e4ad-489b-b42f-f53232be9311\" (UID: \"7d67e149-e4ad-489b-b42f-f53232be9311\") " Oct 31 01:30:07.354440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.354416 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d67e149-e4ad-489b-b42f-f53232be9311-plugins-config-volume" pod "7d67e149-e4ad-489b-b42f-f53232be9311" (UID: "7d67e149-e4ad-489b-b42f-f53232be9311"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:07.355976 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.355954 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d67e149-e4ad-489b-b42f-f53232be9311-kube-api-access-ncpm9" pod "7d67e149-e4ad-489b-b42f-f53232be9311" (UID: "7d67e149-e4ad-489b-b42f-f53232be9311"). InnerVolumeSpecName "kube-api-access-ncpm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:07.455078 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.455042 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncpm9\" (UniqueName: \"kubernetes.io/projected/7d67e149-e4ad-489b-b42f-f53232be9311-kube-api-access-ncpm9\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:07.455078 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.455075 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d67e149-e4ad-489b-b42f-f53232be9311-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:07.961042 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.961013 5890 generic.go:356] "Generic (PLEG): container finished" podID="7d67e149-e4ad-489b-b42f-f53232be9311" containerID="5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df" exitCode=0 Oct 31 01:30:07.961446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.961055 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" Oct 31 01:30:07.961446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.961090 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" event={"ID":"7d67e149-e4ad-489b-b42f-f53232be9311","Type":"ContainerDied","Data":"5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df"} Oct 31 01:30:07.961446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.961141 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl" event={"ID":"7d67e149-e4ad-489b-b42f-f53232be9311","Type":"ContainerDied","Data":"89e8ab67363c09bfbf7c37f453abac085d7f6f0e3a500de3d1e70842f75e1211"} Oct 31 01:30:07.961446 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.961173 5890 scope.go:122] "RemoveContainer" containerID="5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df" Oct 31 01:30:07.963710 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.963181 5890 generic.go:356] "Generic (PLEG): container finished" podID="99739e84-48b2-485e-9cc3-4eca5e4fb179" containerID="e8d87c70bd886f4cb64c2749df34783969a0d32f254b0ac6f7d8fa1d7c0bcfbf" exitCode=0 Oct 31 01:30:07.963710 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.963302 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" event={"ID":"99739e84-48b2-485e-9cc3-4eca5e4fb179","Type":"ContainerDied","Data":"e8d87c70bd886f4cb64c2749df34783969a0d32f254b0ac6f7d8fa1d7c0bcfbf"} Oct 31 01:30:07.965174 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.965154 5890 generic.go:356] "Generic (PLEG): container finished" podID="1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" containerID="3008249e0d68bb6e0bd19f88355a7b0fc12d2213e37e81209af3f9513755bde3" exitCode=0 Oct 31 01:30:07.965285 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.965189 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" event={"ID":"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1","Type":"ContainerDied","Data":"3008249e0d68bb6e0bd19f88355a7b0fc12d2213e37e81209af3f9513755bde3"} Oct 31 01:30:07.971223 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.971194 5890 scope.go:122] "RemoveContainer" containerID="5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df" Oct 31 01:30:07.971491 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:07.971471 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df\": container with ID starting with 5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df not found: ID does not exist" containerID="5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df" Oct 31 01:30:07.971572 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:07.971504 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df"} err="failed to get container status \"5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df\": rpc error: code = NotFound desc = could not find container \"5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df\": container with ID starting with 5e317de2aa4f091f77780f985c68c6abe3842b87b8e565a97e8cc2a0f186d7df not found: ID does not exist" Oct 31 01:30:08.230204 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:08.230111 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl"] Oct 31 01:30:08.234591 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:08.234561 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b63c24-584548d76f-mcbrl"] Oct 31 01:30:09.127249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.127221 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:09.171696 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.171672 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:09.171886 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.171867 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-kube-api-access-hkfdv\" (UniqueName: \"kubernetes.io/projected/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-kube-api-access-hkfdv\") pod \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " Oct 31 01:30:09.171941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.171916 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-xks-ca\") pod \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " Oct 31 01:30:09.171994 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.171982 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-openshift-ca\") pod \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\" (UID: \"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1\") " Oct 31 01:30:09.172357 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.172332 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-openshift-ca" pod "1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" (UID: "1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:09.173958 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.173939 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-kube-api-access-hkfdv" pod "1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" (UID: "1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1"). InnerVolumeSpecName "kube-api-access-hkfdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:09.174033 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.173959 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-xks-ca" pod "1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" (UID: "1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:09.272768 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.272675 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99739e84-48b2-485e-9cc3-4eca5e4fb179-kube-api-access-kwcjn\" (UniqueName: \"kubernetes.io/projected/99739e84-48b2-485e-9cc3-4eca5e4fb179-kube-api-access-kwcjn\") pod \"99739e84-48b2-485e-9cc3-4eca5e4fb179\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " Oct 31 01:30:09.272964 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.272763 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-openshift-ca\") pod \"99739e84-48b2-485e-9cc3-4eca5e4fb179\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " Oct 31 01:30:09.272964 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.272926 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-xks-ca\") pod \"99739e84-48b2-485e-9cc3-4eca5e4fb179\" (UID: \"99739e84-48b2-485e-9cc3-4eca5e4fb179\") " Oct 31 01:30:09.273144 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.273115 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-openshift-ca" pod "99739e84-48b2-485e-9cc3-4eca5e4fb179" (UID: "99739e84-48b2-485e-9cc3-4eca5e4fb179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:09.273340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.273322 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfdv\" (UniqueName: \"kubernetes.io/projected/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-kube-api-access-hkfdv\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.273430 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.273342 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-xks-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.273430 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.273352 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1-maas-api-openshift-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.273430 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.273362 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-openshift-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.275541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.275514 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99739e84-48b2-485e-9cc3-4eca5e4fb179-kube-api-access-kwcjn" pod "99739e84-48b2-485e-9cc3-4eca5e4fb179" (UID: "99739e84-48b2-485e-9cc3-4eca5e4fb179"). InnerVolumeSpecName "kube-api-access-kwcjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:09.275541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.275512 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-xks-ca" pod "99739e84-48b2-485e-9cc3-4eca5e4fb179" (UID: "99739e84-48b2-485e-9cc3-4eca5e4fb179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:09.374347 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.374315 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcjn\" (UniqueName: \"kubernetes.io/projected/99739e84-48b2-485e-9cc3-4eca5e4fb179-kube-api-access-kwcjn\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.374347 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.374344 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99739e84-48b2-485e-9cc3-4eca5e4fb179-maas-api-xks-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.932799 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.932752 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d67e149-e4ad-489b-b42f-f53232be9311" path="/var/lib/kubelet/pods/7d67e149-e4ad-489b-b42f-f53232be9311/volumes" Oct 31 01:30:09.983045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.983011 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" Oct 31 01:30:09.983210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.983020 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv" event={"ID":"1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1","Type":"ContainerDied","Data":"779693db9198aa4cb780ec5dcd45134fb15b9475ad5b19e347a962b9947afafc"} Oct 31 01:30:09.983210 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.983178 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="779693db9198aa4cb780ec5dcd45134fb15b9475ad5b19e347a962b9947afafc" Oct 31 01:30:09.984807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.984757 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" event={"ID":"99739e84-48b2-485e-9cc3-4eca5e4fb179","Type":"ContainerDied","Data":"583c2e8d7231c35b383db78b80cc48ff94816c2d393dafe43804d2eef97b759e"} Oct 31 01:30:09.984807 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.984778 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9" Oct 31 01:30:09.984992 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:09.984842 5890 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="583c2e8d7231c35b383db78b80cc48ff94816c2d393dafe43804d2eef97b759e" Oct 31 01:30:10.968500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:10.968455 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" podUID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:10.968500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:10.968464 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" podUID="71d79d95-9651-4aed-a942-e4d087b335d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:14.926283 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:14.926254 5890 scope.go:122] "RemoveContainer" containerID="99f34ec51bf7e24682b0876705d0206b4f0dce3247f56bfa94d8077203b1e199" Oct 31 01:30:15.774514 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.774477 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj"] Oct 31 01:30:15.775148 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775078 5890 cpu_manager.go:414] "containerMap: removing container" podUID="99739e84-48b2-485e-9cc3-4eca5e4fb179" containerName="revoke-keys" Oct 31 01:30:15.775148 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775107 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99739e84-48b2-485e-9cc3-4eca5e4fb179" containerName="revoke-keys" Oct 31 01:30:15.775495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775177 5890 cpu_manager.go:414] "containerMap: removing container" podUID="7d67e149-e4ad-489b-b42f-f53232be9311" containerName="payload-processing" Oct 31 01:30:15.775495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775187 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d67e149-e4ad-489b-b42f-f53232be9311" containerName="payload-processing" Oct 31 01:30:15.775495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775217 5890 cpu_manager.go:414] "containerMap: removing container" podUID="1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" containerName="revoke-keys" Oct 31 01:30:15.775495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775225 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" containerName="revoke-keys" Oct 31 01:30:15.775495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775359 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d67e149-e4ad-489b-b42f-f53232be9311" containerName="payload-processing" Oct 31 01:30:15.775495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775379 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" containerName="revoke-keys" Oct 31 01:30:15.775495 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.775387 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="99739e84-48b2-485e-9cc3-4eca5e4fb179" containerName="revoke-keys" Oct 31 01:30:15.778961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.778938 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.829217 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.829179 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6bd7c8-serving-cert\"" Oct 31 01:30:15.853823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.852772 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636e6616-e2a9-4113-8c62-11799d3ea892-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.853823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.852878 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-metrics-certs\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.853823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.852988 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/636e6616-e2a9-4113-8c62-11799d3ea892-kube-api-access-2kv9r\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.853823 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.853147 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.879809 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.879730 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj"] Oct 31 01:30:15.953917 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.953878 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636e6616-e2a9-4113-8c62-11799d3ea892-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.954339 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.953926 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-metrics-certs\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.954339 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.953959 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/636e6616-e2a9-4113-8c62-11799d3ea892-kube-api-access-2kv9r\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.954339 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.954056 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.954339 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:15.954180 5890 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-serving-cert: secret "maas-api-e2e-extmod-b-6bd7c8-serving-cert" not found Oct 31 01:30:15.954339 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:15.954257 5890 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls podName:636e6616-e2a9-4113-8c62-11799d3ea892 nodeName:}" failed. No retries permitted until 2026-10-31 01:30:16.454236208 +0000 UTC m=+3708.656486624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls") pod "maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" (UID: "636e6616-e2a9-4113-8c62-11799d3ea892") : secret "maas-api-e2e-extmod-b-6bd7c8-serving-cert" not found Oct 31 01:30:15.954691 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.954667 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636e6616-e2a9-4113-8c62-11799d3ea892-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:15.956684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:15.956655 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-metrics-certs\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:16.020804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.020705 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/636e6616-e2a9-4113-8c62-11799d3ea892-kube-api-access-2kv9r\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:16.032341 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.032311 5890 generic.go:356] "Generic (PLEG): container finished" podID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerID="b8dd11888fa35d306a2f64d9d779f63bf6b05ba32c043e47b637cc966ccd33f1" exitCode=60 Oct 31 01:30:16.032500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.032367 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" event={"ID":"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2","Type":"ContainerDied","Data":"b8dd11888fa35d306a2f64d9d779f63bf6b05ba32c043e47b637cc966ccd33f1"} Oct 31 01:30:16.032500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.032415 5890 scope.go:122] "RemoveContainer" containerID="99f34ec51bf7e24682b0876705d0206b4f0dce3247f56bfa94d8077203b1e199" Oct 31 01:30:16.032840 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.032815 5890 scope.go:122] "RemoveContainer" containerID="b8dd11888fa35d306a2f64d9d779f63bf6b05ba32c043e47b637cc966ccd33f1" Oct 31 01:30:16.033133 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:16.033104 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd_odh-ai-gateway-infra(cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" Oct 31 01:30:16.226464 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.226432 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 01:30:16.349297 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.349267 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 01:30:16.359167 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.359134 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-kube-api-access-vhjqg\" (UniqueName: \"kubernetes.io/projected/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-kube-api-access-vhjqg\") pod \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " Oct 31 01:30:16.359364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.359336 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-plugins-config-volume\") pod \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\" (UID: \"3c4e2839-34eb-4d91-aaa2-969f5dea68d8\") " Oct 31 01:30:16.359711 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.359688 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-plugins-config-volume" pod "3c4e2839-34eb-4d91-aaa2-969f5dea68d8" (UID: "3c4e2839-34eb-4d91-aaa2-969f5dea68d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:16.359948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.359930 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:16.361348 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.361328 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-kube-api-access-vhjqg" pod "3c4e2839-34eb-4d91-aaa2-969f5dea68d8" (UID: "3c4e2839-34eb-4d91-aaa2-969f5dea68d8"). InnerVolumeSpecName "kube-api-access-vhjqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:16.460676 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.460626 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d79d95-9651-4aed-a942-e4d087b335d5-kube-api-access-7hh4d\" (UniqueName: \"kubernetes.io/projected/71d79d95-9651-4aed-a942-e4d087b335d5-kube-api-access-7hh4d\") pod \"71d79d95-9651-4aed-a942-e4d087b335d5\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " Oct 31 01:30:16.460893 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.460751 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d79d95-9651-4aed-a942-e4d087b335d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d79d95-9651-4aed-a942-e4d087b335d5-plugins-config-volume\") pod \"71d79d95-9651-4aed-a942-e4d087b335d5\" (UID: \"71d79d95-9651-4aed-a942-e4d087b335d5\") " Oct 31 01:30:16.461104 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.461077 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:16.461212 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.461183 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d79d95-9651-4aed-a942-e4d087b335d5-plugins-config-volume" pod "71d79d95-9651-4aed-a942-e4d087b335d5" (UID: "71d79d95-9651-4aed-a942-e4d087b335d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:16.461308 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.461290 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d79d95-9651-4aed-a942-e4d087b335d5-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:16.461364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.461314 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjqg\" (UniqueName: \"kubernetes.io/projected/3c4e2839-34eb-4d91-aaa2-969f5dea68d8-kube-api-access-vhjqg\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:16.463566 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.463540 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d79d95-9651-4aed-a942-e4d087b335d5-kube-api-access-7hh4d" pod "71d79d95-9651-4aed-a942-e4d087b335d5" (UID: "71d79d95-9651-4aed-a942-e4d087b335d5"). InnerVolumeSpecName "kube-api-access-7hh4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:16.463661 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.463584 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:16.562539 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.562502 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hh4d\" (UniqueName: \"kubernetes.io/projected/71d79d95-9651-4aed-a942-e4d087b335d5-kube-api-access-7hh4d\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:16.739320 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.739281 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:16.925833 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:16.925771 5890 scope.go:122] "RemoveContainer" containerID="55f44378c2d086c8b5285727da56b2520ae27d7c446bf0b38027f272ec6db8a1" Oct 31 01:30:17.041974 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.041880 5890 generic.go:356] "Generic (PLEG): container finished" podID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" containerID="5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1" exitCode=0 Oct 31 01:30:17.041974 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.041932 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" Oct 31 01:30:17.041974 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.041956 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" event={"ID":"3c4e2839-34eb-4d91-aaa2-969f5dea68d8","Type":"ContainerDied","Data":"5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1"} Oct 31 01:30:17.042537 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.042024 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf" event={"ID":"3c4e2839-34eb-4d91-aaa2-969f5dea68d8","Type":"ContainerDied","Data":"5eac4c743a80733f7e542ed6c08727b177c9afe064f20f723dfc2e7bd1800288"} Oct 31 01:30:17.042537 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.042057 5890 scope.go:122] "RemoveContainer" containerID="5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1" Oct 31 01:30:17.043603 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.043578 5890 generic.go:356] "Generic (PLEG): container finished" podID="71d79d95-9651-4aed-a942-e4d087b335d5" containerID="c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b" exitCode=0 Oct 31 01:30:17.043732 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.043636 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" Oct 31 01:30:17.043732 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.043656 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" event={"ID":"71d79d95-9651-4aed-a942-e4d087b335d5","Type":"ContainerDied","Data":"c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b"} Oct 31 01:30:17.043732 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.043712 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f" event={"ID":"71d79d95-9651-4aed-a942-e4d087b335d5","Type":"ContainerDied","Data":"f11962105be5960017d5867e6fe07dbcbf33a4c00938d267d088662d5113d860"} Oct 31 01:30:17.052754 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.052727 5890 scope.go:122] "RemoveContainer" containerID="5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1" Oct 31 01:30:17.053180 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:17.053099 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1\": container with ID starting with 5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1 not found: ID does not exist" containerID="5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1" Oct 31 01:30:17.053180 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.053137 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1"} err="failed to get container status \"5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1\": rpc error: code = NotFound desc = could not find container \"5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1\": container with ID starting with 5bee9bdb9277b71315a5ade3ac437d36a9d3aa8d2b5ffcc65f07c8c6b077a6f1 not found: ID does not exist" Oct 31 01:30:17.053180 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.053163 5890 scope.go:122] "RemoveContainer" containerID="c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b" Oct 31 01:30:17.063729 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.063704 5890 scope.go:122] "RemoveContainer" containerID="c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b" Oct 31 01:30:17.064064 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:17.064038 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b\": container with ID starting with c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b not found: ID does not exist" containerID="c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b" Oct 31 01:30:17.064152 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.064070 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b"} err="failed to get container status \"c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b\": rpc error: code = NotFound desc = could not find container \"c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b\": container with ID starting with c91ade479d46a2956e91a59f3bbec463c3a79e39d03884404dfa719831854b6b not found: ID does not exist" Oct 31 01:30:17.117325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.117291 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj"] Oct 31 01:30:17.118694 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:30:17.118663 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636e6616_e2a9_4113_8c62_11799d3ea892.slice/crio-5eb386c8a4aa817e171bcdb9c5e10bb4c2b243577d1310ef55210d257d07f751 WatchSource:0}: Error finding container 5eb386c8a4aa817e171bcdb9c5e10bb4c2b243577d1310ef55210d257d07f751: Status 404 returned error can't find the container with id 5eb386c8a4aa817e171bcdb9c5e10bb4c2b243577d1310ef55210d257d07f751 Oct 31 01:30:17.155624 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.155591 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd"] Oct 31 01:30:17.206897 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.206840 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf"] Oct 31 01:30:17.208652 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.208629 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-29ece5-d7c776d9-4jlcf"] Oct 31 01:30:17.309621 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.309553 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:17.371584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.371547 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f"] Oct 31 01:30:17.371818 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.371552 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-maas-api-service-ca\") pod \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " Oct 31 01:30:17.371932 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.371871 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-maas-api-service-ca" pod "cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" (UID: "cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:17.371980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.371929 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-kube-api-access-b8z6k\" (UniqueName: \"kubernetes.io/projected/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-kube-api-access-b8z6k\") pod \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\" (UID: \"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2\") " Oct 31 01:30:17.372242 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.372229 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:17.373993 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.373968 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-kube-api-access-b8z6k" pod "cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" (UID: "cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2"). InnerVolumeSpecName "kube-api-access-b8z6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:17.390709 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.390683 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-29ece5-77b547645f-td54f"] Oct 31 01:30:17.473925 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.473887 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8z6k\" (UniqueName: \"kubernetes.io/projected/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2-kube-api-access-b8z6k\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:17.932749 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.932709 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" path="/var/lib/kubelet/pods/3c4e2839-34eb-4d91-aaa2-969f5dea68d8/volumes" Oct 31 01:30:17.933111 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:17.933092 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d79d95-9651-4aed-a942-e4d087b335d5" path="/var/lib/kubelet/pods/71d79d95-9651-4aed-a942-e4d087b335d5/volumes" Oct 31 01:30:18.052627 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.052591 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" event={"ID":"636e6616-e2a9-4113-8c62-11799d3ea892","Type":"ContainerStarted","Data":"5eb386c8a4aa817e171bcdb9c5e10bb4c2b243577d1310ef55210d257d07f751"} Oct 31 01:30:18.054866 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.054840 5890 generic.go:356] "Generic (PLEG): container finished" podID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerID="5875f6a20d3b975d9075006582ab548a043af012de6a1becc846691fdcdbc5c1" exitCode=60 Oct 31 01:30:18.055010 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.054915 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" event={"ID":"e078e70d-eeea-437a-994a-0a8a0c68f831","Type":"ContainerDied","Data":"5875f6a20d3b975d9075006582ab548a043af012de6a1becc846691fdcdbc5c1"} Oct 31 01:30:18.055010 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.054981 5890 scope.go:122] "RemoveContainer" containerID="55f44378c2d086c8b5285727da56b2520ae27d7c446bf0b38027f272ec6db8a1" Oct 31 01:30:18.055435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.055409 5890 scope.go:122] "RemoveContainer" containerID="5875f6a20d3b975d9075006582ab548a043af012de6a1becc846691fdcdbc5c1" Oct 31 01:30:18.055751 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:18.055727 5890 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf_odh-ai-gateway-infra(e078e70d-eeea-437a-994a-0a8a0c68f831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" Oct 31 01:30:18.056432 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.056410 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" event={"ID":"cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2","Type":"ContainerDied","Data":"15680b1992824da83c437983898d1bd870ecef90d4bbb3e07ecfd7c7644dc0b8"} Oct 31 01:30:18.056517 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.056500 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd" Oct 31 01:30:18.068254 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.068238 5890 scope.go:122] "RemoveContainer" containerID="b8dd11888fa35d306a2f64d9d779f63bf6b05ba32c043e47b637cc966ccd33f1" Oct 31 01:30:18.287171 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.287119 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd"] Oct 31 01:30:18.289721 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:18.289688 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890170-2mqtd"] Oct 31 01:30:19.071660 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.071618 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" event={"ID":"636e6616-e2a9-4113-8c62-11799d3ea892","Type":"ContainerStarted","Data":"2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02"} Oct 31 01:30:19.072208 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.071720 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:19.186721 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.186664 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" podStartSLOduration=3.118722287 podStartE2EDuration="4.186651399s" totalImagesPullingTime="1.067929112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:19.185524095 +0000 UTC m=+3711.387774529" watchObservedRunningTime="2026-10-31 01:30:19.186651399 +0000 UTC m=+3711.388901832" Oct 31 01:30:19.236022 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.235977 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf"] Oct 31 01:30:19.397503 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.397472 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:19.506188 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.506146 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e078e70d-eeea-437a-994a-0a8a0c68f831-kube-api-access-thrj6\" (UniqueName: \"kubernetes.io/projected/e078e70d-eeea-437a-994a-0a8a0c68f831-kube-api-access-thrj6\") pod \"e078e70d-eeea-437a-994a-0a8a0c68f831\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " Oct 31 01:30:19.506374 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.506236 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e078e70d-eeea-437a-994a-0a8a0c68f831-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e078e70d-eeea-437a-994a-0a8a0c68f831-maas-api-service-ca\") pod \"e078e70d-eeea-437a-994a-0a8a0c68f831\" (UID: \"e078e70d-eeea-437a-994a-0a8a0c68f831\") " Oct 31 01:30:19.506629 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.506604 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e078e70d-eeea-437a-994a-0a8a0c68f831-maas-api-service-ca" pod "e078e70d-eeea-437a-994a-0a8a0c68f831" (UID: "e078e70d-eeea-437a-994a-0a8a0c68f831"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:19.506721 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.506702 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e078e70d-eeea-437a-994a-0a8a0c68f831-maas-api-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.508587 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.508561 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e078e70d-eeea-437a-994a-0a8a0c68f831-kube-api-access-thrj6" pod "e078e70d-eeea-437a-994a-0a8a0c68f831" (UID: "e078e70d-eeea-437a-994a-0a8a0c68f831"). InnerVolumeSpecName "kube-api-access-thrj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:19.607715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.607675 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thrj6\" (UniqueName: \"kubernetes.io/projected/e078e70d-eeea-437a-994a-0a8a0c68f831-kube-api-access-thrj6\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.933881 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:19.933847 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" path="/var/lib/kubelet/pods/cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2/volumes" Oct 31 01:30:20.080008 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:20.079980 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" Oct 31 01:30:20.080502 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:20.080027 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf" event={"ID":"e078e70d-eeea-437a-994a-0a8a0c68f831","Type":"ContainerDied","Data":"3ad170cd21bfe3a41ea5e3bcaa2be8b29edac5de44859385a92dced09eaa81d4"} Oct 31 01:30:20.080502 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:20.080101 5890 scope.go:122] "RemoveContainer" containerID="5875f6a20d3b975d9075006582ab548a043af012de6a1becc846691fdcdbc5c1" Oct 31 01:30:20.241691 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:20.241608 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf"] Oct 31 01:30:20.289052 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:20.289021 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890170-rdmdf"] Oct 31 01:30:21.938399 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:21.938364 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" path="/var/lib/kubelet/pods/e078e70d-eeea-437a-994a-0a8a0c68f831/volumes" Oct 31 01:30:25.086404 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.086367 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:30:25.329396 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.329359 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk"] Oct 31 01:30:25.329656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.329621 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" podUID="a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" containerName="maas-api" containerID="cri-o://2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7" gracePeriod=30 Oct 31 01:30:25.609157 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.609095 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:30:25.782486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.782449 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-metrics-certs\") pod \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " Oct 31 01:30:25.782687 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.782540 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls\") pod \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " Oct 31 01:30:25.782687 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.782650 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-openshift-service-ca\") pod \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " Oct 31 01:30:25.782687 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.782683 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-kube-api-access-g4hrf\" (UniqueName: \"kubernetes.io/projected/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-kube-api-access-g4hrf\") pod \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\" (UID: \"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af\") " Oct 31 01:30:25.783116 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.783075 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-openshift-service-ca" pod "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" (UID: "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:25.784954 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.784923 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-kube-api-access-g4hrf" pod "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" (UID: "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af"). InnerVolumeSpecName "kube-api-access-g4hrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:25.785206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.785170 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls" pod "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" (UID: "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:25.785315 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.785239 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-metrics-certs" pod "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" (UID: "a6f4edaf-4c2c-4345-9a20-61c27b2fc9af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:25.884035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.883953 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:25.884035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.883983 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:25.884035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.883993 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:25.884035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:25.884005 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4hrf\" (UniqueName: \"kubernetes.io/projected/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af-kube-api-access-g4hrf\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:26.126474 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.126436 5890 generic.go:356] "Generic (PLEG): container finished" podID="a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" containerID="2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7" exitCode=0 Oct 31 01:30:26.126939 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.126484 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" Oct 31 01:30:26.126939 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.126522 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" event={"ID":"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af","Type":"ContainerDied","Data":"2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7"} Oct 31 01:30:26.126939 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.126577 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk" event={"ID":"a6f4edaf-4c2c-4345-9a20-61c27b2fc9af","Type":"ContainerDied","Data":"cc0fe472892d9b957231667acd92625cf36160dd620b95646dc33ab058c8b2d8"} Oct 31 01:30:26.126939 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.126600 5890 scope.go:122] "RemoveContainer" containerID="2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7" Oct 31 01:30:26.136336 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.136308 5890 scope.go:122] "RemoveContainer" containerID="2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7" Oct 31 01:30:26.136603 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:26.136584 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7\": container with ID starting with 2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7 not found: ID does not exist" containerID="2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7" Oct 31 01:30:26.136663 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.136612 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7"} err="failed to get container status \"2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7\": rpc error: code = NotFound desc = could not find container \"2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7\": container with ID starting with 2c5f63720cca7526404bc438ca72134e934f58ee8838561f6e5540719e7520e7 not found: ID does not exist" Oct 31 01:30:26.285757 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.285724 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk"] Oct 31 01:30:26.332584 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:26.332545 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a72754-7ff947cf6c-psnkk"] Oct 31 01:30:27.932878 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:27.932849 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" path="/var/lib/kubelet/pods/a6f4edaf-4c2c-4345-9a20-61c27b2fc9af/volumes" Oct 31 01:30:37.020362 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:30:37.020318 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:30:47.902216 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:47.902175 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r"] Oct 31 01:30:48.010813 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:48.010763 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc"] Oct 31 01:30:52.903094 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:52.903047 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" podUID="b6b0b7a9-4e13-42a7-809f-360661bb960e" containerName="payload-processing" containerID="cri-o://464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2" gracePeriod=25 Oct 31 01:30:53.011947 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:53.011906 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" podUID="8a6b1212-8008-4f0e-a86f-638da400aeea" containerName="payload-pre-processing" containerID="cri-o://4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00" gracePeriod=25 Oct 31 01:30:54.069595 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:54.069562 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" podUID="8a6b1212-8008-4f0e-a86f-638da400aeea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:54.070045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:30:54.069553 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" podUID="b6b0b7a9-4e13-42a7-809f-360661bb960e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:31:04.070060 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:04.070019 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" podUID="b6b0b7a9-4e13-42a7-809f-360661bb960e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:31:04.070659 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:04.070026 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" podUID="8a6b1212-8008-4f0e-a86f-638da400aeea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:31:07.018232 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:07.018182 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:31:13.180796 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.180755 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 01:31:13.268004 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.267982 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 01:31:13.294325 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.294292 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b0b7a9-4e13-42a7-809f-360661bb960e-kube-api-access-zl2td\" (UniqueName: \"kubernetes.io/projected/b6b0b7a9-4e13-42a7-809f-360661bb960e-kube-api-access-zl2td\") pod \"b6b0b7a9-4e13-42a7-809f-360661bb960e\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " Oct 31 01:31:13.294475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.294462 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b0b7a9-4e13-42a7-809f-360661bb960e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b0b7a9-4e13-42a7-809f-360661bb960e-plugins-config-volume\") pod \"b6b0b7a9-4e13-42a7-809f-360661bb960e\" (UID: \"b6b0b7a9-4e13-42a7-809f-360661bb960e\") " Oct 31 01:31:13.294833 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.294806 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b0b7a9-4e13-42a7-809f-360661bb960e-plugins-config-volume" pod "b6b0b7a9-4e13-42a7-809f-360661bb960e" (UID: "b6b0b7a9-4e13-42a7-809f-360661bb960e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:13.296497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.296472 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b0b7a9-4e13-42a7-809f-360661bb960e-kube-api-access-zl2td" pod "b6b0b7a9-4e13-42a7-809f-360661bb960e" (UID: "b6b0b7a9-4e13-42a7-809f-360661bb960e"). InnerVolumeSpecName "kube-api-access-zl2td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:13.395414 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.395373 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6b1212-8008-4f0e-a86f-638da400aeea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b1212-8008-4f0e-a86f-638da400aeea-plugins-config-volume\") pod \"8a6b1212-8008-4f0e-a86f-638da400aeea\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " Oct 31 01:31:13.395566 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.395425 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6b1212-8008-4f0e-a86f-638da400aeea-kube-api-access-4llxc\" (UniqueName: \"kubernetes.io/projected/8a6b1212-8008-4f0e-a86f-638da400aeea-kube-api-access-4llxc\") pod \"8a6b1212-8008-4f0e-a86f-638da400aeea\" (UID: \"8a6b1212-8008-4f0e-a86f-638da400aeea\") " Oct 31 01:31:13.395712 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.395696 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b0b7a9-4e13-42a7-809f-360661bb960e-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:13.395820 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.395712 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl2td\" (UniqueName: \"kubernetes.io/projected/b6b0b7a9-4e13-42a7-809f-360661bb960e-kube-api-access-zl2td\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:13.395820 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.395720 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6b1212-8008-4f0e-a86f-638da400aeea-plugins-config-volume" pod "8a6b1212-8008-4f0e-a86f-638da400aeea" (UID: "8a6b1212-8008-4f0e-a86f-638da400aeea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:13.397519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.397497 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6b1212-8008-4f0e-a86f-638da400aeea-kube-api-access-4llxc" pod "8a6b1212-8008-4f0e-a86f-638da400aeea" (UID: "8a6b1212-8008-4f0e-a86f-638da400aeea"). InnerVolumeSpecName "kube-api-access-4llxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:13.426822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.426772 5890 generic.go:356] "Generic (PLEG): container finished" podID="8a6b1212-8008-4f0e-a86f-638da400aeea" containerID="4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00" exitCode=0 Oct 31 01:31:13.426980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.426834 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" Oct 31 01:31:13.426980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.426824 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" event={"ID":"8a6b1212-8008-4f0e-a86f-638da400aeea","Type":"ContainerDied","Data":"4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00"} Oct 31 01:31:13.426980 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.426968 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc" event={"ID":"8a6b1212-8008-4f0e-a86f-638da400aeea","Type":"ContainerDied","Data":"8942e3edf8d1780528fba5c57087cf05391e29fa4d051f1465a52e68e3461df5"} Oct 31 01:31:13.427184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.427001 5890 scope.go:122] "RemoveContainer" containerID="4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00" Oct 31 01:31:13.428939 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.428914 5890 generic.go:356] "Generic (PLEG): container finished" podID="b6b0b7a9-4e13-42a7-809f-360661bb960e" containerID="464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2" exitCode=0 Oct 31 01:31:13.429077 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.428964 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" event={"ID":"b6b0b7a9-4e13-42a7-809f-360661bb960e","Type":"ContainerDied","Data":"464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2"} Oct 31 01:31:13.429077 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.429005 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" event={"ID":"b6b0b7a9-4e13-42a7-809f-360661bb960e","Type":"ContainerDied","Data":"48287d6ed431252dcea87f05d292701f643fb78018767a3607a83eb9ca81bf98"} Oct 31 01:31:13.429213 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.429076 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r" Oct 31 01:31:13.440412 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.440314 5890 scope.go:122] "RemoveContainer" containerID="4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00" Oct 31 01:31:13.440759 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:13.440733 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00\": container with ID starting with 4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00 not found: ID does not exist" containerID="4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00" Oct 31 01:31:13.441086 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.441056 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00"} err="failed to get container status \"4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00\": rpc error: code = NotFound desc = could not find container \"4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00\": container with ID starting with 4d60d3f000964d73d12dbcc4549ab72957e951d5abe54f8e83f1905e1efeeb00 not found: ID does not exist" Oct 31 01:31:13.441086 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.441086 5890 scope.go:122] "RemoveContainer" containerID="464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2" Oct 31 01:31:13.450863 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.450825 5890 scope.go:122] "RemoveContainer" containerID="464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2" Oct 31 01:31:13.451190 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:13.451165 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2\": container with ID starting with 464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2 not found: ID does not exist" containerID="464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2" Oct 31 01:31:13.451262 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.451200 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2"} err="failed to get container status \"464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2\": rpc error: code = NotFound desc = could not find container \"464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2\": container with ID starting with 464fde2f60425b9e8d77b945298c3b23b3e316854f9e98186a37d0cecb1e5cc2 not found: ID does not exist" Oct 31 01:31:13.496984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.496957 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6b1212-8008-4f0e-a86f-638da400aeea-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:13.496984 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.496984 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4llxc\" (UniqueName: \"kubernetes.io/projected/8a6b1212-8008-4f0e-a86f-638da400aeea-kube-api-access-4llxc\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:13.607358 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.607324 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc"] Oct 31 01:31:13.611967 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.611939 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ef2572-66ff5d84b7-q78cc"] Oct 31 01:31:13.717721 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.717641 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r"] Oct 31 01:31:13.721343 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.721310 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ef2572-6ddf767845-8mv7r"] Oct 31 01:31:13.932221 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.932190 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6b1212-8008-4f0e-a86f-638da400aeea" path="/var/lib/kubelet/pods/8a6b1212-8008-4f0e-a86f-638da400aeea/volumes" Oct 31 01:31:13.932694 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:13.932674 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b0b7a9-4e13-42a7-809f-360661bb960e" path="/var/lib/kubelet/pods/b6b0b7a9-4e13-42a7-809f-360661bb960e/volumes" Oct 31 01:31:28.861840 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:28.861801 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57"] Oct 31 01:31:28.862455 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:28.862027 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" podUID="2498dc56-4e7a-45f0-bc55-ce80a12e774b" containerName="maas-api" containerID="cri-o://717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4" gracePeriod=30 Oct 31 01:31:29.133773 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.133747 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 01:31:29.261289 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.261256 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2498dc56-4e7a-45f0-bc55-ce80a12e774b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2498dc56-4e7a-45f0-bc55-ce80a12e774b-openshift-service-ca\") pod \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " Oct 31 01:31:29.261462 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.261322 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-metrics-certs\") pod \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " Oct 31 01:31:29.261462 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.261378 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2498dc56-4e7a-45f0-bc55-ce80a12e774b-kube-api-access-jbhhs\" (UniqueName: \"kubernetes.io/projected/2498dc56-4e7a-45f0-bc55-ce80a12e774b-kube-api-access-jbhhs\") pod \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " Oct 31 01:31:29.261462 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.261427 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls\") pod \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\" (UID: \"2498dc56-4e7a-45f0-bc55-ce80a12e774b\") " Oct 31 01:31:29.261641 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.261617 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2498dc56-4e7a-45f0-bc55-ce80a12e774b-openshift-service-ca" pod "2498dc56-4e7a-45f0-bc55-ce80a12e774b" (UID: "2498dc56-4e7a-45f0-bc55-ce80a12e774b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:29.261842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.261821 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2498dc56-4e7a-45f0-bc55-ce80a12e774b-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:29.263634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.263608 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls" pod "2498dc56-4e7a-45f0-bc55-ce80a12e774b" (UID: "2498dc56-4e7a-45f0-bc55-ce80a12e774b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:29.263634 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.263626 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-metrics-certs" pod "2498dc56-4e7a-45f0-bc55-ce80a12e774b" (UID: "2498dc56-4e7a-45f0-bc55-ce80a12e774b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:29.263819 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.263651 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2498dc56-4e7a-45f0-bc55-ce80a12e774b-kube-api-access-jbhhs" pod "2498dc56-4e7a-45f0-bc55-ce80a12e774b" (UID: "2498dc56-4e7a-45f0-bc55-ce80a12e774b"). InnerVolumeSpecName "kube-api-access-jbhhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:29.363486 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.363445 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:29.363648 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.363485 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhhs\" (UniqueName: \"kubernetes.io/projected/2498dc56-4e7a-45f0-bc55-ce80a12e774b-kube-api-access-jbhhs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:29.363648 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.363515 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2498dc56-4e7a-45f0-bc55-ce80a12e774b-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:29.528369 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.528332 5890 generic.go:356] "Generic (PLEG): container finished" podID="2498dc56-4e7a-45f0-bc55-ce80a12e774b" containerID="717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4" exitCode=0 Oct 31 01:31:29.528551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.528385 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" Oct 31 01:31:29.528551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.528391 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" event={"ID":"2498dc56-4e7a-45f0-bc55-ce80a12e774b","Type":"ContainerDied","Data":"717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4"} Oct 31 01:31:29.528551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.528502 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57" event={"ID":"2498dc56-4e7a-45f0-bc55-ce80a12e774b","Type":"ContainerDied","Data":"41524ee90e97ddd92522cadf32a1d2f6b5842d15042b3c736ba850b502a80700"} Oct 31 01:31:29.528551 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.528523 5890 scope.go:122] "RemoveContainer" containerID="717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4" Oct 31 01:31:29.539033 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.539013 5890 scope.go:122] "RemoveContainer" containerID="717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4" Oct 31 01:31:29.539316 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:29.539296 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4\": container with ID starting with 717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4 not found: ID does not exist" containerID="717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4" Oct 31 01:31:29.539389 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.539325 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4"} err="failed to get container status \"717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4\": rpc error: code = NotFound desc = could not find container \"717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4\": container with ID starting with 717e5a7e7b0781c68ccbc869c839b7b154b2ed04cd5b3428576f7223541d93c4 not found: ID does not exist" Oct 31 01:31:29.690405 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.690363 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57"] Oct 31 01:31:29.693383 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.693357 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ef2572-594d87bc54-mtg57"] Oct 31 01:31:29.933982 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:29.933941 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2498dc56-4e7a-45f0-bc55-ce80a12e774b" path="/var/lib/kubelet/pods/2498dc56-4e7a-45f0-bc55-ce80a12e774b/volumes" Oct 31 01:31:37.021839 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:37.021739 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:31:40.083308 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:40.083270 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl"] Oct 31 01:31:40.193814 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:40.193768 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9"] Oct 31 01:31:45.083701 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.083655 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" podUID="2a114a39-0af2-437e-8b9e-1353ae7b7267" containerName="payload-processing" containerID="cri-o://9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f" gracePeriod=25 Oct 31 01:31:45.111885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.111854 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg"] Oct 31 01:31:45.112117 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.112074 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" podUID="9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" containerName="maas-api" containerID="cri-o://efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9" gracePeriod=30 Oct 31 01:31:45.385332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.385298 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:31:45.415909 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.415884 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:31:45.422934 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.422913 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a114a39-0af2-437e-8b9e-1353ae7b7267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a114a39-0af2-437e-8b9e-1353ae7b7267-plugins-config-volume\") pod \"2a114a39-0af2-437e-8b9e-1353ae7b7267\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " Oct 31 01:31:45.423015 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.422965 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a114a39-0af2-437e-8b9e-1353ae7b7267-kube-api-access-n68qx\" (UniqueName: \"kubernetes.io/projected/2a114a39-0af2-437e-8b9e-1353ae7b7267-kube-api-access-n68qx\") pod \"2a114a39-0af2-437e-8b9e-1353ae7b7267\" (UID: \"2a114a39-0af2-437e-8b9e-1353ae7b7267\") " Oct 31 01:31:45.423266 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.423247 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a114a39-0af2-437e-8b9e-1353ae7b7267-plugins-config-volume" pod "2a114a39-0af2-437e-8b9e-1353ae7b7267" (UID: "2a114a39-0af2-437e-8b9e-1353ae7b7267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:45.423374 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.423359 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a114a39-0af2-437e-8b9e-1353ae7b7267-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:45.424989 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.424971 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a114a39-0af2-437e-8b9e-1353ae7b7267-kube-api-access-n68qx" pod "2a114a39-0af2-437e-8b9e-1353ae7b7267" (UID: "2a114a39-0af2-437e-8b9e-1353ae7b7267"). InnerVolumeSpecName "kube-api-access-n68qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:45.523691 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.523663 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-kube-api-access-pj8n2\" (UniqueName: \"kubernetes.io/projected/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-kube-api-access-pj8n2\") pod \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " Oct 31 01:31:45.523872 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.523701 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-openshift-service-ca\") pod \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " Oct 31 01:31:45.523872 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.523853 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-metrics-certs\") pod \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " Oct 31 01:31:45.523971 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.523895 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls\") pod \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\" (UID: \"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1\") " Oct 31 01:31:45.524111 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.524085 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68qx\" (UniqueName: \"kubernetes.io/projected/2a114a39-0af2-437e-8b9e-1353ae7b7267-kube-api-access-n68qx\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:45.524252 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.524155 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-openshift-service-ca" pod "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" (UID: "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:45.525955 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.525925 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-kube-api-access-pj8n2" pod "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" (UID: "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1"). InnerVolumeSpecName "kube-api-access-pj8n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:45.526084 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.526056 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls" pod "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" (UID: "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:45.526460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.526437 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-metrics-certs" pod "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" (UID: "9b545ee4-8ec4-4f82-8e1e-182d95a48bc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:45.593193 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.593081 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" podUID="e1879df4-734d-4002-9fff-4b5c1c1128e7" containerName="payload-pre-processing" containerID="cri-o://d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8" gracePeriod=25 Oct 31 01:31:45.624997 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.624955 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj8n2\" (UniqueName: \"kubernetes.io/projected/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-kube-api-access-pj8n2\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:45.624997 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.624997 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:45.625176 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.625014 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:45.625176 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.625029 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:45.626298 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.626271 5890 generic.go:356] "Generic (PLEG): container finished" podID="2a114a39-0af2-437e-8b9e-1353ae7b7267" containerID="9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f" exitCode=0 Oct 31 01:31:45.626432 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.626319 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" Oct 31 01:31:45.626432 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.626350 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" event={"ID":"2a114a39-0af2-437e-8b9e-1353ae7b7267","Type":"ContainerDied","Data":"9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f"} Oct 31 01:31:45.626432 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.626406 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl" event={"ID":"2a114a39-0af2-437e-8b9e-1353ae7b7267","Type":"ContainerDied","Data":"6694860fec8e0b558ad62cb9a9587dc493f83b4a6d675fae320a580589dc3a0b"} Oct 31 01:31:45.626605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.626437 5890 scope.go:122] "RemoveContainer" containerID="9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f" Oct 31 01:31:45.628075 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.628050 5890 generic.go:356] "Generic (PLEG): container finished" podID="9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" containerID="efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9" exitCode=0 Oct 31 01:31:45.628204 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.628097 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" event={"ID":"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1","Type":"ContainerDied","Data":"efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9"} Oct 31 01:31:45.628204 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.628117 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" Oct 31 01:31:45.628204 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.628155 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg" event={"ID":"9b545ee4-8ec4-4f82-8e1e-182d95a48bc1","Type":"ContainerDied","Data":"1664da182994745b7a8d53d4d1c65593558ed1d584bd9e3075295a7dc0038692"} Oct 31 01:31:45.705148 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.705126 5890 scope.go:122] "RemoveContainer" containerID="9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f" Oct 31 01:31:45.705498 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:45.705475 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f\": container with ID starting with 9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f not found: ID does not exist" containerID="9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f" Oct 31 01:31:45.705605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.705505 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f"} err="failed to get container status \"9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f\": rpc error: code = NotFound desc = could not find container \"9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f\": container with ID starting with 9cd7a401374e9732f3979b53f72deeb63e7d89145f1d957e9a295789ff52320f not found: ID does not exist" Oct 31 01:31:45.705605 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.705524 5890 scope.go:122] "RemoveContainer" containerID="efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9" Oct 31 01:31:45.714364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.714342 5890 scope.go:122] "RemoveContainer" containerID="efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9" Oct 31 01:31:45.714647 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:45.714622 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9\": container with ID starting with efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9 not found: ID does not exist" containerID="efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9" Oct 31 01:31:45.714798 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.714650 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9"} err="failed to get container status \"efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9\": rpc error: code = NotFound desc = could not find container \"efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9\": container with ID starting with efafb038caec0eeedd92d56e4eeae84411bc14d5903b1e1b1622afe3f2ab72f9 not found: ID does not exist" Oct 31 01:31:45.787184 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.787146 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl"] Oct 31 01:31:45.791044 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.791014 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c61da9-78dcc86799-qspwl"] Oct 31 01:31:45.855428 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.855371 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:31:45.894159 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.894122 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg"] Oct 31 01:31:45.898712 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.898685 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c61da9-5cd8fb899d-t6zzg"] Oct 31 01:31:45.929249 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.929210 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1879df4-734d-4002-9fff-4b5c1c1128e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1879df4-734d-4002-9fff-4b5c1c1128e7-plugins-config-volume\") pod \"e1879df4-734d-4002-9fff-4b5c1c1128e7\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " Oct 31 01:31:45.929405 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.929302 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1879df4-734d-4002-9fff-4b5c1c1128e7-kube-api-access-ckf54\" (UniqueName: \"kubernetes.io/projected/e1879df4-734d-4002-9fff-4b5c1c1128e7-kube-api-access-ckf54\") pod \"e1879df4-734d-4002-9fff-4b5c1c1128e7\" (UID: \"e1879df4-734d-4002-9fff-4b5c1c1128e7\") " Oct 31 01:31:45.931672 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.931640 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1879df4-734d-4002-9fff-4b5c1c1128e7-plugins-config-volume" pod "e1879df4-734d-4002-9fff-4b5c1c1128e7" (UID: "e1879df4-734d-4002-9fff-4b5c1c1128e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:45.932580 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.932543 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1879df4-734d-4002-9fff-4b5c1c1128e7-kube-api-access-ckf54" pod "e1879df4-734d-4002-9fff-4b5c1c1128e7" (UID: "e1879df4-734d-4002-9fff-4b5c1c1128e7"). InnerVolumeSpecName "kube-api-access-ckf54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:45.937164 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.937133 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a114a39-0af2-437e-8b9e-1353ae7b7267" path="/var/lib/kubelet/pods/2a114a39-0af2-437e-8b9e-1353ae7b7267/volumes" Oct 31 01:31:45.937772 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:45.937749 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" path="/var/lib/kubelet/pods/9b545ee4-8ec4-4f82-8e1e-182d95a48bc1/volumes" Oct 31 01:31:46.030843 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.030809 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1879df4-734d-4002-9fff-4b5c1c1128e7-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:46.030843 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.030841 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckf54\" (UniqueName: \"kubernetes.io/projected/e1879df4-734d-4002-9fff-4b5c1c1128e7-kube-api-access-ckf54\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:46.636923 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.636885 5890 generic.go:356] "Generic (PLEG): container finished" podID="e1879df4-734d-4002-9fff-4b5c1c1128e7" containerID="d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8" exitCode=0 Oct 31 01:31:46.637395 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.636959 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" Oct 31 01:31:46.637395 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.636964 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" event={"ID":"e1879df4-734d-4002-9fff-4b5c1c1128e7","Type":"ContainerDied","Data":"d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8"} Oct 31 01:31:46.637395 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.637030 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9" event={"ID":"e1879df4-734d-4002-9fff-4b5c1c1128e7","Type":"ContainerDied","Data":"d2eb77911ba302a1d3bafcd332dfad53bbd45d0566b89de403162019c1c6ef3b"} Oct 31 01:31:46.637395 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.637060 5890 scope.go:122] "RemoveContainer" containerID="d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8" Oct 31 01:31:46.646499 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.646479 5890 scope.go:122] "RemoveContainer" containerID="d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8" Oct 31 01:31:46.646746 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:31:46.646728 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8\": container with ID starting with d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8 not found: ID does not exist" containerID="d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8" Oct 31 01:31:46.646816 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.646753 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8"} err="failed to get container status \"d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8\": rpc error: code = NotFound desc = could not find container \"d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8\": container with ID starting with d252650c64f0231743022f6d302191b40316509b9648751ed02f83741e350fa8 not found: ID does not exist" Oct 31 01:31:46.794610 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.794575 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9"] Oct 31 01:31:46.801377 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:46.801352 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c61da9-6f87f9496c-96cr9"] Oct 31 01:31:47.932202 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:47.932158 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1879df4-734d-4002-9fff-4b5c1c1128e7" path="/var/lib/kubelet/pods/e1879df4-734d-4002-9fff-4b5c1c1128e7/volumes" Oct 31 01:31:54.560452 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:54.560415 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll"] Oct 31 01:31:54.561086 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:54.560697 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" containerID="cri-o://2736fd97e93a33c6566e8b1a4629f01a728404b0f7883d03e0751a0590a6d95b" gracePeriod=30 Oct 31 01:31:59.716306 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.716060 5890 generic.go:356] "Generic (PLEG): container finished" podID="42be952b-15bf-41c7-b416-3f259d97849b" containerID="2736fd97e93a33c6566e8b1a4629f01a728404b0f7883d03e0751a0590a6d95b" exitCode=0 Oct 31 01:31:59.716306 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.716145 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" event={"ID":"42be952b-15bf-41c7-b416-3f259d97849b","Type":"ContainerDied","Data":"2736fd97e93a33c6566e8b1a4629f01a728404b0f7883d03e0751a0590a6d95b"} Oct 31 01:31:59.716306 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.716275 5890 scope.go:122] "RemoveContainer" containerID="c027315a48fb1c90ab4423beb4d93f7fee0d632c1550f8b5b26470c6baae8659" Oct 31 01:31:59.832981 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.832920 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:31:59.976069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976030 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-istio-token\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-istio-token\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976098 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-socket\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976124 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-certs\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976154 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-data\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976205 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42be952b-15bf-41c7-b416-3f259d97849b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42be952b-15bf-41c7-b416-3f259d97849b-istio-podinfo\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976238 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42be952b-15bf-41c7-b416-3f259d97849b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42be952b-15bf-41c7-b416-3f259d97849b-istiod-ca-cert\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976277 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-kube-api-access-vbq86\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-kube-api-access-vbq86\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976309 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-credential-socket\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976398 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-envoy\") pod \"42be952b-15bf-41c7-b416-3f259d97849b\" (UID: \"42be952b-15bf-41c7-b416-3f259d97849b\") " Oct 31 01:31:59.976540 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976444 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-socket" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:59.976761 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976600 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-data" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:59.976954 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976929 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:59.977033 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.976960 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:59.977188 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.977167 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-credential-socket" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:59.979137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.978038 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42be952b-15bf-41c7-b416-3f259d97849b-istiod-ca-cert" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:59.979137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.978105 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-certs" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:59.979137 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.979065 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-envoy" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:59.979453 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.979428 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-istio-token" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:59.980277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.980237 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-kube-api-access-vbq86" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "kube-api-access-vbq86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:59.980277 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:31:59.980264 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42be952b-15bf-41c7-b416-3f259d97849b-istio-podinfo" pod "42be952b-15bf-41c7-b416-3f259d97849b" (UID: "42be952b-15bf-41c7-b416-3f259d97849b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:32:00.078519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.078484 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:00.078519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.078515 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:00.078519 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.078525 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42be952b-15bf-41c7-b416-3f259d97849b-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:00.078753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.078535 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42be952b-15bf-41c7-b416-3f259d97849b-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:00.078753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.078546 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbq86\" (UniqueName: \"kubernetes.io/projected/42be952b-15bf-41c7-b416-3f259d97849b-kube-api-access-vbq86\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:00.078753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.078558 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:00.078753 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.078567 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42be952b-15bf-41c7-b416-3f259d97849b-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:00.723639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.723603 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" Oct 31 01:32:00.723639 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.723614 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll" event={"ID":"42be952b-15bf-41c7-b416-3f259d97849b","Type":"ContainerDied","Data":"beddc595869fd6e60998fe815a88aff2a3cd8d96880d103cad209a4317278e6f"} Oct 31 01:32:00.724215 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.723674 5890 scope.go:122] "RemoveContainer" containerID="2736fd97e93a33c6566e8b1a4629f01a728404b0f7883d03e0751a0590a6d95b" Oct 31 01:32:00.929118 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.929083 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll"] Oct 31 01:32:00.977020 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:00.976926 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c61da9-openshift-default-76f5864d45-n9kll"] Oct 31 01:32:01.932396 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:01.932353 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42be952b-15bf-41c7-b416-3f259d97849b" path="/var/lib/kubelet/pods/42be952b-15bf-41c7-b416-3f259d97849b/volumes" Oct 31 01:32:07.018250 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:32:07.018212 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:32:07.105366 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:07.105326 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9"] Oct 31 01:32:07.219774 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:07.219734 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p"] Oct 31 01:32:12.105983 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.105932 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" podUID="4899c6c7-5566-4f4c-adb6-9e6c696d4838" containerName="payload-processing" containerID="cri-o://68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb" gracePeriod=25 Oct 31 01:32:12.220163 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.220123 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" podUID="8c4268ea-a56f-424c-b547-0bf580665e33" containerName="payload-pre-processing" containerID="cri-o://58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd" gracePeriod=25 Oct 31 01:32:12.381818 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.381764 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:32:12.408932 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.408899 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4899c6c7-5566-4f4c-adb6-9e6c696d4838-kube-api-access-9kd6l\" (UniqueName: \"kubernetes.io/projected/4899c6c7-5566-4f4c-adb6-9e6c696d4838-kube-api-access-9kd6l\") pod \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " Oct 31 01:32:12.409101 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.408970 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4899c6c7-5566-4f4c-adb6-9e6c696d4838-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4899c6c7-5566-4f4c-adb6-9e6c696d4838-plugins-config-volume\") pod \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\" (UID: \"4899c6c7-5566-4f4c-adb6-9e6c696d4838\") " Oct 31 01:32:12.409388 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.409368 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4899c6c7-5566-4f4c-adb6-9e6c696d4838-plugins-config-volume" pod "4899c6c7-5566-4f4c-adb6-9e6c696d4838" (UID: "4899c6c7-5566-4f4c-adb6-9e6c696d4838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:12.411209 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.411174 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4899c6c7-5566-4f4c-adb6-9e6c696d4838-kube-api-access-9kd6l" pod "4899c6c7-5566-4f4c-adb6-9e6c696d4838" (UID: "4899c6c7-5566-4f4c-adb6-9e6c696d4838"). InnerVolumeSpecName "kube-api-access-9kd6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:12.474513 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.474484 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:32:12.510251 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.510217 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4268ea-a56f-424c-b547-0bf580665e33-kube-api-access-hj82r\" (UniqueName: \"kubernetes.io/projected/8c4268ea-a56f-424c-b547-0bf580665e33-kube-api-access-hj82r\") pod \"8c4268ea-a56f-424c-b547-0bf580665e33\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " Oct 31 01:32:12.510449 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.510308 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4268ea-a56f-424c-b547-0bf580665e33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4268ea-a56f-424c-b547-0bf580665e33-plugins-config-volume\") pod \"8c4268ea-a56f-424c-b547-0bf580665e33\" (UID: \"8c4268ea-a56f-424c-b547-0bf580665e33\") " Oct 31 01:32:12.510601 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.510581 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kd6l\" (UniqueName: \"kubernetes.io/projected/4899c6c7-5566-4f4c-adb6-9e6c696d4838-kube-api-access-9kd6l\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:12.510671 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.510607 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4899c6c7-5566-4f4c-adb6-9e6c696d4838-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:12.510742 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.510715 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4268ea-a56f-424c-b547-0bf580665e33-plugins-config-volume" pod "8c4268ea-a56f-424c-b547-0bf580665e33" (UID: "8c4268ea-a56f-424c-b547-0bf580665e33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:12.512497 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.512464 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4268ea-a56f-424c-b547-0bf580665e33-kube-api-access-hj82r" pod "8c4268ea-a56f-424c-b547-0bf580665e33" (UID: "8c4268ea-a56f-424c-b547-0bf580665e33"). InnerVolumeSpecName "kube-api-access-hj82r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:12.611804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.611695 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj82r\" (UniqueName: \"kubernetes.io/projected/8c4268ea-a56f-424c-b547-0bf580665e33-kube-api-access-hj82r\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:12.611804 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.611733 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4268ea-a56f-424c-b547-0bf580665e33-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:12.800350 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.800314 5890 generic.go:356] "Generic (PLEG): container finished" podID="4899c6c7-5566-4f4c-adb6-9e6c696d4838" containerID="68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb" exitCode=0 Oct 31 01:32:12.800350 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.800359 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" Oct 31 01:32:12.800602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.800404 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" event={"ID":"4899c6c7-5566-4f4c-adb6-9e6c696d4838","Type":"ContainerDied","Data":"68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb"} Oct 31 01:32:12.800602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.800470 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9" event={"ID":"4899c6c7-5566-4f4c-adb6-9e6c696d4838","Type":"ContainerDied","Data":"8ad05e06f10036dc952af1585f68592e6c30200e34515c515f94c8fcfd88a114"} Oct 31 01:32:12.800602 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.800500 5890 scope.go:122] "RemoveContainer" containerID="68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb" Oct 31 01:32:12.801920 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.801893 5890 generic.go:356] "Generic (PLEG): container finished" podID="8c4268ea-a56f-424c-b547-0bf580665e33" containerID="58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd" exitCode=0 Oct 31 01:32:12.802040 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.801954 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" event={"ID":"8c4268ea-a56f-424c-b547-0bf580665e33","Type":"ContainerDied","Data":"58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd"} Oct 31 01:32:12.802040 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.802006 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" event={"ID":"8c4268ea-a56f-424c-b547-0bf580665e33","Type":"ContainerDied","Data":"34bae8353854ce909708be071e28f0d0695d361b13ffc26f59e9e60ddcb356b4"} Oct 31 01:32:12.802155 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.802137 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p" Oct 31 01:32:12.812024 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.812008 5890 scope.go:122] "RemoveContainer" containerID="68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb" Oct 31 01:32:12.812320 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:32:12.812292 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb\": container with ID starting with 68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb not found: ID does not exist" containerID="68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb" Oct 31 01:32:12.812408 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.812331 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb"} err="failed to get container status \"68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb\": rpc error: code = NotFound desc = could not find container \"68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb\": container with ID starting with 68eb7b15f0e4d14e25787cd2486357fc67d9444e8ad76fa7ef4db75f8077deeb not found: ID does not exist" Oct 31 01:32:12.812408 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.812358 5890 scope.go:122] "RemoveContainer" containerID="58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd" Oct 31 01:32:12.821151 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.821136 5890 scope.go:122] "RemoveContainer" containerID="58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd" Oct 31 01:32:12.821400 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:32:12.821384 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd\": container with ID starting with 58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd not found: ID does not exist" containerID="58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd" Oct 31 01:32:12.821440 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.821408 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd"} err="failed to get container status \"58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd\": rpc error: code = NotFound desc = could not find container \"58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd\": container with ID starting with 58b0a411602f5a1432eee3539c7e1ad52cd5591c218b101688d8b1a7cc91ccdd not found: ID does not exist" Oct 31 01:32:12.957795 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.957741 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9"] Oct 31 01:32:12.962206 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:12.962177 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6bd7c8-7c8f784fd9-mn6j9"] Oct 31 01:32:13.066637 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:13.066601 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p"] Oct 31 01:32:13.071692 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:13.071664 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6bd7c8-58894cfb4c-7wb2p"] Oct 31 01:32:13.931652 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:13.931616 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4899c6c7-5566-4f4c-adb6-9e6c696d4838" path="/var/lib/kubelet/pods/4899c6c7-5566-4f4c-adb6-9e6c696d4838/volumes" Oct 31 01:32:13.932069 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:13.932004 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4268ea-a56f-424c-b547-0bf580665e33" path="/var/lib/kubelet/pods/8c4268ea-a56f-424c-b547-0bf580665e33/volumes" Oct 31 01:32:16.057549 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.057518 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj"] Oct 31 01:32:16.057975 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.057741 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" podUID="636e6616-e2a9-4113-8c62-11799d3ea892" containerName="maas-api" containerID="cri-o://2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02" gracePeriod=30 Oct 31 01:32:16.324889 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.324832 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:32:16.348579 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.348551 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636e6616-e2a9-4113-8c62-11799d3ea892-kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/636e6616-e2a9-4113-8c62-11799d3ea892-kube-api-access-2kv9r\") pod \"636e6616-e2a9-4113-8c62-11799d3ea892\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " Oct 31 01:32:16.348730 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.348606 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls\") pod \"636e6616-e2a9-4113-8c62-11799d3ea892\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " Oct 31 01:32:16.348730 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.348667 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636e6616-e2a9-4113-8c62-11799d3ea892-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636e6616-e2a9-4113-8c62-11799d3ea892-openshift-service-ca\") pod \"636e6616-e2a9-4113-8c62-11799d3ea892\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " Oct 31 01:32:16.348891 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.348743 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-metrics-certs\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-metrics-certs\") pod \"636e6616-e2a9-4113-8c62-11799d3ea892\" (UID: \"636e6616-e2a9-4113-8c62-11799d3ea892\") " Oct 31 01:32:16.349103 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.349080 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636e6616-e2a9-4113-8c62-11799d3ea892-openshift-service-ca" pod "636e6616-e2a9-4113-8c62-11799d3ea892" (UID: "636e6616-e2a9-4113-8c62-11799d3ea892"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:16.350750 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.350723 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636e6616-e2a9-4113-8c62-11799d3ea892-kube-api-access-2kv9r" pod "636e6616-e2a9-4113-8c62-11799d3ea892" (UID: "636e6616-e2a9-4113-8c62-11799d3ea892"). InnerVolumeSpecName "kube-api-access-2kv9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:16.350933 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.350751 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls" pod "636e6616-e2a9-4113-8c62-11799d3ea892" (UID: "636e6616-e2a9-4113-8c62-11799d3ea892"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:16.350933 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.350795 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-metrics-certs" pod "636e6616-e2a9-4113-8c62-11799d3ea892" (UID: "636e6616-e2a9-4113-8c62-11799d3ea892"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:16.450041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.450005 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636e6616-e2a9-4113-8c62-11799d3ea892-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:16.450041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.450037 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:16.450041 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.450048 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kv9r\" (UniqueName: \"kubernetes.io/projected/636e6616-e2a9-4113-8c62-11799d3ea892-kube-api-access-2kv9r\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:16.450305 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.450062 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636e6616-e2a9-4113-8c62-11799d3ea892-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:16.827658 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.827617 5890 generic.go:356] "Generic (PLEG): container finished" podID="636e6616-e2a9-4113-8c62-11799d3ea892" containerID="2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02" exitCode=0 Oct 31 01:32:16.827851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.827671 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" Oct 31 01:32:16.827851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.827705 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" event={"ID":"636e6616-e2a9-4113-8c62-11799d3ea892","Type":"ContainerDied","Data":"2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02"} Oct 31 01:32:16.827851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.827799 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj" event={"ID":"636e6616-e2a9-4113-8c62-11799d3ea892","Type":"ContainerDied","Data":"5eb386c8a4aa817e171bcdb9c5e10bb4c2b243577d1310ef55210d257d07f751"} Oct 31 01:32:16.827851 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.827821 5890 scope.go:122] "RemoveContainer" containerID="2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02" Oct 31 01:32:16.838932 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.838906 5890 scope.go:122] "RemoveContainer" containerID="2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02" Oct 31 01:32:16.839213 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:32:16.839192 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02\": container with ID starting with 2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02 not found: ID does not exist" containerID="2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02" Oct 31 01:32:16.839282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.839227 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02"} err="failed to get container status \"2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02\": rpc error: code = NotFound desc = could not find container \"2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02\": container with ID starting with 2144425304dc5f022fac7e7968232615e89b106ff6c80006d764855ccc464b02 not found: ID does not exist" Oct 31 01:32:16.986550 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.986509 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj"] Oct 31 01:32:16.989967 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:16.989931 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6bd7c8-6dbb45fb49-l7jqj"] Oct 31 01:32:17.932734 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:17.932699 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636e6616-e2a9-4113-8c62-11799d3ea892" path="/var/lib/kubelet/pods/636e6616-e2a9-4113-8c62-11799d3ea892/volumes" Oct 31 01:32:37.018255 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:32:37.018210 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:32:37.755494 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:37.755456 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt"] Oct 31 01:32:42.756327 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:42.756286 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" podUID="dc16ba85-0f91-4224-8257-2b27b920a7d3" containerName="payload-pre-processing" containerID="cri-o://3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473" gracePeriod=25 Oct 31 01:32:45.428916 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:45.428877 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" podUID="dc16ba85-0f91-4224-8257-2b27b920a7d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:32:50.106743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:50.106706 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84"] Oct 31 01:32:50.107247 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:50.106941 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" containerID="cri-o://26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305" gracePeriod=30 Oct 31 01:32:55.364417 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.364390 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:32:55.428836 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.428767 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" podUID="dc16ba85-0f91-4224-8257-2b27b920a7d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:32:55.538022 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.537985 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-socket\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538022 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538027 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-certs\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538125 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00e0b1d-36be-4089-96a1-f35924e0fda9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00e0b1d-36be-4089-96a1-f35924e0fda9-istiod-ca-cert\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538175 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-podinfo\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538217 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-kube-api-access-9q4sz\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-kube-api-access-9q4sz\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538272 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538262 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-data\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538290 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-token\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-token\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538322 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-socket" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:55.538483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538332 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-envoy\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538364 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-credential-socket\") pod \"b00e0b1d-36be-4089-96a1-f35924e0fda9\" (UID: \"b00e0b1d-36be-4089-96a1-f35924e0fda9\") " Oct 31 01:32:55.538483 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538401 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-certs" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:55.538733 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538480 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00e0b1d-36be-4089-96a1-f35924e0fda9-istiod-ca-cert" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:55.538733 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538552 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-data" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:55.538733 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538665 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-credential-socket" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:55.538926 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538797 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.538926 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538817 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.538926 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538829 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.538926 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538845 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00e0b1d-36be-4089-96a1-f35924e0fda9-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.538926 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.538865 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.540656 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.540622 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-podinfo" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:32:55.540775 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.540743 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-token" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:55.543193 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.543169 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-envoy" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:55.543307 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.543187 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-kube-api-access-9q4sz" pod "b00e0b1d-36be-4089-96a1-f35924e0fda9" (UID: "b00e0b1d-36be-4089-96a1-f35924e0fda9"). InnerVolumeSpecName "kube-api-access-9q4sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:55.639961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.639926 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.639961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.639954 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.639961 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.639964 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00e0b1d-36be-4089-96a1-f35924e0fda9-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.640205 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:55.639975 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q4sz\" (UniqueName: \"kubernetes.io/projected/b00e0b1d-36be-4089-96a1-f35924e0fda9-kube-api-access-9q4sz\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:56.063614 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.063578 5890 generic.go:356] "Generic (PLEG): container finished" podID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerID="26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305" exitCode=0 Oct 31 01:32:56.063822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.063635 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" Oct 31 01:32:56.063822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.063659 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" event={"ID":"b00e0b1d-36be-4089-96a1-f35924e0fda9","Type":"ContainerDied","Data":"26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305"} Oct 31 01:32:56.063822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.063721 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84" event={"ID":"b00e0b1d-36be-4089-96a1-f35924e0fda9","Type":"ContainerDied","Data":"260f95519c91de163ee1b39e6464e6508dec02bcacba13a701fc6ebf6c918d7a"} Oct 31 01:32:56.063822 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.063755 5890 scope.go:122] "RemoveContainer" containerID="26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305" Oct 31 01:32:56.073177 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.073155 5890 scope.go:122] "RemoveContainer" containerID="daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00" Oct 31 01:32:56.081726 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.081702 5890 scope.go:122] "RemoveContainer" containerID="26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305" Oct 31 01:32:56.082000 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:32:56.081976 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305\": container with ID starting with 26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305 not found: ID does not exist" containerID="26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305" Oct 31 01:32:56.082074 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.082009 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305"} err="failed to get container status \"26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305\": rpc error: code = NotFound desc = could not find container \"26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305\": container with ID starting with 26c23131ddb5419c4b70d629225145fcf3c89c99abfd7dbe68a6b84fd6ad9305 not found: ID does not exist" Oct 31 01:32:56.082074 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.082027 5890 scope.go:122] "RemoveContainer" containerID="daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00" Oct 31 01:32:56.082300 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:32:56.082282 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00\": container with ID starting with daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00 not found: ID does not exist" containerID="daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00" Oct 31 01:32:56.082349 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.082307 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00"} err="failed to get container status \"daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00\": rpc error: code = NotFound desc = could not find container \"daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00\": container with ID starting with daa934657f1aa096445c8705018eb5c252137aa469593788c94a8f0620473b00 not found: ID does not exist" Oct 31 01:32:56.270665 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.270586 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84"] Oct 31 01:32:56.318053 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:56.318020 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cb9a07-openshift-default-5cf9754dc9-7pw84"] Oct 31 01:32:57.932768 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:32:57.932733 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" path="/var/lib/kubelet/pods/b00e0b1d-36be-4089-96a1-f35924e0fda9/volumes" Oct 31 01:33:03.015765 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.015702 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:33:03.108051 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.108018 5890 generic.go:356] "Generic (PLEG): container finished" podID="dc16ba85-0f91-4224-8257-2b27b920a7d3" containerID="3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473" exitCode=0 Oct 31 01:33:03.108239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.108073 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" Oct 31 01:33:03.108239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.108095 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" event={"ID":"dc16ba85-0f91-4224-8257-2b27b920a7d3","Type":"ContainerDied","Data":"3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473"} Oct 31 01:33:03.108239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.108147 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt" event={"ID":"dc16ba85-0f91-4224-8257-2b27b920a7d3","Type":"ContainerDied","Data":"b422bfdd2c93bb46750c5cb32ef5d27f1d41e897ecaf9a4b46a53f211aab1a85"} Oct 31 01:33:03.108239 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.108170 5890 scope.go:122] "RemoveContainer" containerID="3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473" Oct 31 01:33:03.109364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.109346 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc16ba85-0f91-4224-8257-2b27b920a7d3-kube-api-access-b6846\" (UniqueName: \"kubernetes.io/projected/dc16ba85-0f91-4224-8257-2b27b920a7d3-kube-api-access-b6846\") pod \"dc16ba85-0f91-4224-8257-2b27b920a7d3\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " Oct 31 01:33:03.109460 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.109435 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc16ba85-0f91-4224-8257-2b27b920a7d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc16ba85-0f91-4224-8257-2b27b920a7d3-plugins-config-volume\") pod \"dc16ba85-0f91-4224-8257-2b27b920a7d3\" (UID: \"dc16ba85-0f91-4224-8257-2b27b920a7d3\") " Oct 31 01:33:03.109840 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.109817 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc16ba85-0f91-4224-8257-2b27b920a7d3-plugins-config-volume" pod "dc16ba85-0f91-4224-8257-2b27b920a7d3" (UID: "dc16ba85-0f91-4224-8257-2b27b920a7d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:33:03.111494 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.111472 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc16ba85-0f91-4224-8257-2b27b920a7d3-kube-api-access-b6846" pod "dc16ba85-0f91-4224-8257-2b27b920a7d3" (UID: "dc16ba85-0f91-4224-8257-2b27b920a7d3"). InnerVolumeSpecName "kube-api-access-b6846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:03.124526 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.124508 5890 scope.go:122] "RemoveContainer" containerID="3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473" Oct 31 01:33:03.124836 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:03.124813 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473\": container with ID starting with 3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473 not found: ID does not exist" containerID="3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473" Oct 31 01:33:03.124922 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.124847 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473"} err="failed to get container status \"3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473\": rpc error: code = NotFound desc = could not find container \"3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473\": container with ID starting with 3c52ea39f844a814fe8597962d6074e724b8ff7928a2e2d483892e31e255e473 not found: ID does not exist" Oct 31 01:33:03.210149 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.210119 5890 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc16ba85-0f91-4224-8257-2b27b920a7d3-plugins-config-volume\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:03.210149 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.210146 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6846\" (UniqueName: \"kubernetes.io/projected/dc16ba85-0f91-4224-8257-2b27b920a7d3-kube-api-access-b6846\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:03.525979 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.525941 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt"] Oct 31 01:33:03.527914 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.527884 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cb9a07-546ff77d99-fxjpt"] Oct 31 01:33:03.932024 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:03.931994 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc16ba85-0f91-4224-8257-2b27b920a7d3" path="/var/lib/kubelet/pods/dc16ba85-0f91-4224-8257-2b27b920a7d3/volumes" Oct 31 01:33:07.016872 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:07.016830 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:33:22.676416 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:22.676332 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z"] Oct 31 01:33:22.676996 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:22.676639 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" containerID="cri-o://da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0" gracePeriod=30 Oct 31 01:33:27.940561 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:27.940534 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:33:28.054268 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054183 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-podinfo\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054433 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054311 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-socket\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054433 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054341 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3815179-7ff3-4db0-b671-ecdafd4e977c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3815179-7ff3-4db0-b671-ecdafd4e977c-istiod-ca-cert\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054433 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054375 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-kube-api-access-rcnbq\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-kube-api-access-rcnbq\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054436 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-data\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054473 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-certs\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054502 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-token\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-token\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054541 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-envoy\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054594 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-credential-socket\") pod \"d3815179-7ff3-4db0-b671-ecdafd4e977c\" (UID: \"d3815179-7ff3-4db0-b671-ecdafd4e977c\") " Oct 31 01:33:28.054938 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054645 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-socket" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:28.054938 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054812 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-certs" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:28.055066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054972 5890 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.055066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.054991 5890 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-workload-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.055066 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.055015 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3815179-7ff3-4db0-b671-ecdafd4e977c-istiod-ca-cert" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:33:28.055240 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.055152 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-data" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:28.055339 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.055317 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-credential-socket" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:28.056825 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.056771 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-token" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:28.057088 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.057067 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-kube-api-access-rcnbq" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "kube-api-access-rcnbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:28.057174 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.057098 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-envoy" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:28.057226 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.057171 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-podinfo" pod "d3815179-7ff3-4db0-b671-ecdafd4e977c" (UID: "d3815179-7ff3-4db0-b671-ecdafd4e977c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:33:28.156340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.156309 5890 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-token\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.156340 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.156340 5890 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-envoy\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.156535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.156354 5890 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-credential-socket\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.156535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.156364 5890 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-podinfo\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.156535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.156375 5890 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3815179-7ff3-4db0-b671-ecdafd4e977c-istiod-ca-cert\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.156535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.156384 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcnbq\" (UniqueName: \"kubernetes.io/projected/d3815179-7ff3-4db0-b671-ecdafd4e977c-kube-api-access-rcnbq\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.156535 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.156393 5890 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3815179-7ff3-4db0-b671-ecdafd4e977c-istio-data\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.237208 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.237171 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45"] Oct 31 01:33:28.258102 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.258056 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6"] Oct 31 01:33:28.261801 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.261753 5890 generic.go:356] "Generic (PLEG): container finished" podID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerID="da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0" exitCode=0 Oct 31 01:33:28.262008 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.261985 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" Oct 31 01:33:28.262193 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.261993 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" event={"ID":"d3815179-7ff3-4db0-b671-ecdafd4e977c","Type":"ContainerDied","Data":"da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0"} Oct 31 01:33:28.262285 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.262266 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z" event={"ID":"d3815179-7ff3-4db0-b671-ecdafd4e977c","Type":"ContainerDied","Data":"120e1e49298dcc6a3f43a242856a2ef5726e9f342281be4136e53fc92b497c33"} Oct 31 01:33:28.262330 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.262314 5890 scope.go:122] "RemoveContainer" containerID="da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0" Oct 31 01:33:28.278962 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.278942 5890 scope.go:122] "RemoveContainer" containerID="ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b" Oct 31 01:33:28.288332 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.288309 5890 scope.go:122] "RemoveContainer" containerID="da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0" Oct 31 01:33:28.288646 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:28.288623 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0\": container with ID starting with da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0 not found: ID does not exist" containerID="da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0" Oct 31 01:33:28.288708 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.288655 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0"} err="failed to get container status \"da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0\": rpc error: code = NotFound desc = could not find container \"da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0\": container with ID starting with da0fa4a989ece15ae3286248e4893e40e26fa54ec975a495fada108084c3f0e0 not found: ID does not exist" Oct 31 01:33:28.288708 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.288672 5890 scope.go:122] "RemoveContainer" containerID="ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b" Oct 31 01:33:28.288950 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:28.288932 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b\": container with ID starting with ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b not found: ID does not exist" containerID="ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b" Oct 31 01:33:28.289007 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.288958 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b"} err="failed to get container status \"ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b\": rpc error: code = NotFound desc = could not find container \"ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b\": container with ID starting with ff5d66f51985273103d3e695767aeca3f4e20c69054c0027b76d8ebab0dc803b not found: ID does not exist" Oct 31 01:33:28.619006 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.618967 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z"] Oct 31 01:33:28.624773 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:28.624744 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7f8596-openshift-default-7d6dd886c-6mr6z"] Oct 31 01:33:29.074668 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:29.074642 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:33:29.186554 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:29.186528 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jlcr7_341fbe38-f97c-4a00-938b-6c4b9a30197e/console-operator/1.log" Oct 31 01:33:29.931725 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:29.931689 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" path="/var/lib/kubelet/pods/d3815179-7ff3-4db0-b671-ecdafd4e977c/volumes" Oct 31 01:33:37.017964 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:37.017925 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:33:43.261729 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.261652 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" containerID="cri-o://46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e" gracePeriod=45 Oct 31 01:33:43.287609 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.287569 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" containerID="cri-o://7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8" gracePeriod=45 Oct 31 01:33:43.470282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.470233 5890 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused" Oct 31 01:33:43.559179 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.559148 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 01:33:43.573168 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.573140 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 01:33:43.606706 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606674 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-dshm\") pod \"bab22885-f796-4636-9e98-7622e5a6757a\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " Oct 31 01:33:43.606903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606719 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab22885-f796-4636-9e98-7622e5a6757a-kube-api-access-vjvnx\" (UniqueName: \"kubernetes.io/projected/bab22885-f796-4636-9e98-7622e5a6757a-kube-api-access-vjvnx\") pod \"bab22885-f796-4636-9e98-7622e5a6757a\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " Oct 31 01:33:43.606903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606750 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-model-cache\") pod \"2f176e3c-5114-48dd-880f-2150d4bc5b22\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " Oct 31 01:33:43.606903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606837 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-tmp-dir\") pod \"bab22885-f796-4636-9e98-7622e5a6757a\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " Oct 31 01:33:43.606903 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606865 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-home\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-home\") pod \"bab22885-f796-4636-9e98-7622e5a6757a\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " Oct 31 01:33:43.607229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606903 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-dshm\") pod \"2f176e3c-5114-48dd-880f-2150d4bc5b22\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " Oct 31 01:33:43.607229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606966 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f176e3c-5114-48dd-880f-2150d4bc5b22-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f176e3c-5114-48dd-880f-2150d4bc5b22-tls-certs\") pod \"2f176e3c-5114-48dd-880f-2150d4bc5b22\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " Oct 31 01:33:43.607229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.606997 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-home\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-home\") pod \"2f176e3c-5114-48dd-880f-2150d4bc5b22\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " Oct 31 01:33:43.607229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607023 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-model-cache\") pod \"bab22885-f796-4636-9e98-7622e5a6757a\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " Oct 31 01:33:43.607229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607076 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab22885-f796-4636-9e98-7622e5a6757a-tls-certs\" (UniqueName: \"kubernetes.io/secret/bab22885-f796-4636-9e98-7622e5a6757a-tls-certs\") pod \"bab22885-f796-4636-9e98-7622e5a6757a\" (UID: \"bab22885-f796-4636-9e98-7622e5a6757a\") " Oct 31 01:33:43.607229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607131 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-tmp-dir\") pod \"2f176e3c-5114-48dd-880f-2150d4bc5b22\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " Oct 31 01:33:43.607229 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607169 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f176e3c-5114-48dd-880f-2150d4bc5b22-kube-api-access-5t2ll\" (UniqueName: \"kubernetes.io/projected/2f176e3c-5114-48dd-880f-2150d4bc5b22-kube-api-access-5t2ll\") pod \"2f176e3c-5114-48dd-880f-2150d4bc5b22\" (UID: \"2f176e3c-5114-48dd-880f-2150d4bc5b22\") " Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607073 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-model-cache" pod "2f176e3c-5114-48dd-880f-2150d4bc5b22" (UID: "2f176e3c-5114-48dd-880f-2150d4bc5b22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607303 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-model-cache" pod "bab22885-f796-4636-9e98-7622e5a6757a" (UID: "bab22885-f796-4636-9e98-7622e5a6757a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607433 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-home" pod "2f176e3c-5114-48dd-880f-2150d4bc5b22" (UID: "2f176e3c-5114-48dd-880f-2150d4bc5b22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607545 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607573 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607587 5890 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-model-cache\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607101 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-tmp-dir" pod "bab22885-f796-4636-9e98-7622e5a6757a" (UID: "bab22885-f796-4636-9e98-7622e5a6757a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.607625 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607218 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-home" pod "bab22885-f796-4636-9e98-7622e5a6757a" (UID: "bab22885-f796-4636-9e98-7622e5a6757a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.608090 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.607646 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-tmp-dir" pod "2f176e3c-5114-48dd-880f-2150d4bc5b22" (UID: "2f176e3c-5114-48dd-880f-2150d4bc5b22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.609779 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.609742 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f176e3c-5114-48dd-880f-2150d4bc5b22-tls-certs" pod "2f176e3c-5114-48dd-880f-2150d4bc5b22" (UID: "2f176e3c-5114-48dd-880f-2150d4bc5b22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:43.609923 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.609849 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-dshm" pod "2f176e3c-5114-48dd-880f-2150d4bc5b22" (UID: "2f176e3c-5114-48dd-880f-2150d4bc5b22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.609923 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.609857 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f176e3c-5114-48dd-880f-2150d4bc5b22-kube-api-access-5t2ll" pod "2f176e3c-5114-48dd-880f-2150d4bc5b22" (UID: "2f176e3c-5114-48dd-880f-2150d4bc5b22"). InnerVolumeSpecName "kube-api-access-5t2ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:43.610043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.609957 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-dshm" pod "bab22885-f796-4636-9e98-7622e5a6757a" (UID: "bab22885-f796-4636-9e98-7622e5a6757a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.610043 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.609969 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab22885-f796-4636-9e98-7622e5a6757a-tls-certs" pod "bab22885-f796-4636-9e98-7622e5a6757a" (UID: "bab22885-f796-4636-9e98-7622e5a6757a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:43.610129 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.610067 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab22885-f796-4636-9e98-7622e5a6757a-kube-api-access-vjvnx" pod "bab22885-f796-4636-9e98-7622e5a6757a" (UID: "bab22885-f796-4636-9e98-7622e5a6757a"). InnerVolumeSpecName "kube-api-access-vjvnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:43.708966 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.708933 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.708966 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.708963 5890 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-home\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.708966 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.708972 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.709282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.708980 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f176e3c-5114-48dd-880f-2150d4bc5b22-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.709282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.708991 5890 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab22885-f796-4636-9e98-7622e5a6757a-tls-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.709282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.709000 5890 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f176e3c-5114-48dd-880f-2150d4bc5b22-tmp-dir\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.709282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.709010 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2ll\" (UniqueName: \"kubernetes.io/projected/2f176e3c-5114-48dd-880f-2150d4bc5b22-kube-api-access-5t2ll\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.709282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.709019 5890 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab22885-f796-4636-9e98-7622e5a6757a-dshm\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.709282 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:43.709029 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvnx\" (UniqueName: \"kubernetes.io/projected/bab22885-f796-4636-9e98-7622e5a6757a-kube-api-access-vjvnx\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:44.354256 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.354209 5890 generic.go:356] "Generic (PLEG): container finished" podID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerID="46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e" exitCode=0 Oct 31 01:33:44.354256 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.354260 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" Oct 31 01:33:44.354839 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.354290 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" event={"ID":"2f176e3c-5114-48dd-880f-2150d4bc5b22","Type":"ContainerDied","Data":"46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e"} Oct 31 01:33:44.354839 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.354341 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45" event={"ID":"2f176e3c-5114-48dd-880f-2150d4bc5b22","Type":"ContainerDied","Data":"261e8ea6c98fffe60b1a9389d4d7f9272078ffd3269fdfc2ab00d3624694df9c"} Oct 31 01:33:44.354839 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.354361 5890 scope.go:122] "RemoveContainer" containerID="46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e" Oct 31 01:33:44.355803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.355762 5890 generic.go:356] "Generic (PLEG): container finished" podID="bab22885-f796-4636-9e98-7622e5a6757a" containerID="7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8" exitCode=0 Oct 31 01:33:44.355896 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.355841 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" Oct 31 01:33:44.355952 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.355841 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" event={"ID":"bab22885-f796-4636-9e98-7622e5a6757a","Type":"ContainerDied","Data":"7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8"} Oct 31 01:33:44.356007 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.355980 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6" event={"ID":"bab22885-f796-4636-9e98-7622e5a6757a","Type":"ContainerDied","Data":"824d62c75449be628028b7ba2391dd1e95d6cef8fe5d0aa73d6be69e87843810"} Oct 31 01:33:44.364035 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.364013 5890 scope.go:122] "RemoveContainer" containerID="46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e" Oct 31 01:33:44.364339 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:44.364318 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e\": container with ID starting with 46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e not found: ID does not exist" containerID="46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e" Oct 31 01:33:44.364445 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.364346 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e"} err="failed to get container status \"46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e\": rpc error: code = NotFound desc = could not find container \"46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e\": container with ID starting with 46cd18dc043f468370f0f2c03eabf1db3142632ba5aebf0317d5464f2f4f176e not found: ID does not exist" Oct 31 01:33:44.364445 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.364363 5890 scope.go:122] "RemoveContainer" containerID="7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8" Oct 31 01:33:44.373259 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.373234 5890 scope.go:122] "RemoveContainer" containerID="7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8" Oct 31 01:33:44.373539 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:44.373520 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8\": container with ID starting with 7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8 not found: ID does not exist" containerID="7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8" Oct 31 01:33:44.373614 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.373547 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8"} err="failed to get container status \"7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8\": rpc error: code = NotFound desc = could not find container \"7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8\": container with ID starting with 7ceea548d1a472d262a1c28cdf66734925300c53bee8d4fce59dee9c6fe8c7e8 not found: ID does not exist" Oct 31 01:33:44.516395 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.516361 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45"] Oct 31 01:33:44.563208 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.563130 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-w0-8bec73-kserve-69b5f5c7d5-9rm45"] Oct 31 01:33:44.668453 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.668417 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6"] Oct 31 01:33:44.673138 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:44.673111 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/simulator-w0-8bec73-kserve-5c5d75dbc9-qxpm6"] Oct 31 01:33:45.931488 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:45.931446 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" path="/var/lib/kubelet/pods/2f176e3c-5114-48dd-880f-2150d4bc5b22/volumes" Oct 31 01:33:45.932092 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:45.932071 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab22885-f796-4636-9e98-7622e5a6757a" path="/var/lib/kubelet/pods/bab22885-f796-4636-9e98-7622e5a6757a/volumes" Oct 31 01:33:54.635245 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.635209 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh"] Oct 31 01:33:54.635928 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.635482 5890 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" podUID="c91f41c9-0702-4750-94d7-c660db68c749" containerName="maas-api" containerID="cri-o://cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af" gracePeriod=30 Oct 31 01:33:54.886295 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.886240 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 01:33:54.912374 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.912348 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls\") pod \"c91f41c9-0702-4750-94d7-c660db68c749\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " Oct 31 01:33:54.912511 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.912420 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91f41c9-0702-4750-94d7-c660db68c749-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91f41c9-0702-4750-94d7-c660db68c749-openshift-service-ca\") pod \"c91f41c9-0702-4750-94d7-c660db68c749\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " Oct 31 01:33:54.912511 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.912445 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-metrics-certs\") pod \"c91f41c9-0702-4750-94d7-c660db68c749\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " Oct 31 01:33:54.912511 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.912501 5890 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91f41c9-0702-4750-94d7-c660db68c749-kube-api-access-4d7dq\" (UniqueName: \"kubernetes.io/projected/c91f41c9-0702-4750-94d7-c660db68c749-kube-api-access-4d7dq\") pod \"c91f41c9-0702-4750-94d7-c660db68c749\" (UID: \"c91f41c9-0702-4750-94d7-c660db68c749\") " Oct 31 01:33:54.912755 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.912731 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91f41c9-0702-4750-94d7-c660db68c749-openshift-service-ca" pod "c91f41c9-0702-4750-94d7-c660db68c749" (UID: "c91f41c9-0702-4750-94d7-c660db68c749"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:33:54.914435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.914411 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls" pod "c91f41c9-0702-4750-94d7-c660db68c749" (UID: "c91f41c9-0702-4750-94d7-c660db68c749"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:54.914570 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.914550 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91f41c9-0702-4750-94d7-c660db68c749-kube-api-access-4d7dq" pod "c91f41c9-0702-4750-94d7-c660db68c749" (UID: "c91f41c9-0702-4750-94d7-c660db68c749"). InnerVolumeSpecName "kube-api-access-4d7dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:54.914637 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:54.914570 5890 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-metrics-certs" pod "c91f41c9-0702-4750-94d7-c660db68c749" (UID: "c91f41c9-0702-4750-94d7-c660db68c749"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:55.013948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.013909 5890 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d7dq\" (UniqueName: \"kubernetes.io/projected/c91f41c9-0702-4750-94d7-c660db68c749-kube-api-access-4d7dq\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:55.013948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.013940 5890 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-maas-api-tls\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:55.013948 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.013957 5890 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91f41c9-0702-4750-94d7-c660db68c749-openshift-service-ca\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:55.014216 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.013967 5890 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91f41c9-0702-4750-94d7-c660db68c749-metrics-certs\") on node \"ip-10-0-143-105.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:55.416632 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.416596 5890 generic.go:356] "Generic (PLEG): container finished" podID="c91f41c9-0702-4750-94d7-c660db68c749" containerID="cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af" exitCode=0 Oct 31 01:33:55.416864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.416641 5890 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" Oct 31 01:33:55.416864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.416692 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" event={"ID":"c91f41c9-0702-4750-94d7-c660db68c749","Type":"ContainerDied","Data":"cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af"} Oct 31 01:33:55.416864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.416744 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh" event={"ID":"c91f41c9-0702-4750-94d7-c660db68c749","Type":"ContainerDied","Data":"f72f0275ffe338fce63c612e43f9f35080d9eda9f97b0f34e38a61d8328a8f92"} Oct 31 01:33:55.416864 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.416769 5890 scope.go:122] "RemoveContainer" containerID="cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af" Oct 31 01:33:55.426681 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.426665 5890 scope.go:122] "RemoveContainer" containerID="cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af" Oct 31 01:33:55.426952 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:33:55.426929 5890 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af\": container with ID starting with cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af not found: ID does not exist" containerID="cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af" Oct 31 01:33:55.427028 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.426961 5890 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af"} err="failed to get container status \"cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af\": rpc error: code = NotFound desc = could not find container \"cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af\": container with ID starting with cb45a6e95727ccdfa0fd0629271c9e901025910a49115c414d90417f234043af not found: ID does not exist" Oct 31 01:33:55.575016 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.574981 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh"] Oct 31 01:33:55.579622 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.579593 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-8bec73-78887949fc-rjsvh"] Oct 31 01:33:55.931715 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:33:55.931676 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91f41c9-0702-4750-94d7-c660db68c749" path="/var/lib/kubelet/pods/c91f41c9-0702-4750-94d7-c660db68c749/volumes" Oct 31 01:34:05.766287 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:05.766251 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f7f7dc8cd-g96sp_91127689-97c8-4ca7-82a1-6d2d22f3a9c4/authorino/0.log" Oct 31 01:34:06.206582 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:06.206547 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqmtl_f877144e-05ba-4ead-a809-5daea44d1075/manager/0.log" Oct 31 01:34:06.479547 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:06.479472 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zp68g_d98fb307-575b-4e36-8e52-9398d884733a/manager/0.log" Oct 31 01:34:07.017854 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:34:07.017814 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:34:08.260576 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:08.260531 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6b4pn_8c5a0c2c-a45a-45d0-9e71-505ad5dff953/manager/0.log" Oct 31 01:34:08.777642 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:08.777602 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6k4fb_1fd906c8-7686-4cb4-9057-0d8b1da239d1/manager/0.log" Oct 31 01:34:11.306870 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:11.306838 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:34:12.946951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:12.946922 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f7f7dc8cd-g96sp_91127689-97c8-4ca7-82a1-6d2d22f3a9c4/authorino/0.log" Oct 31 01:34:25.405186 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:25.405155 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:34:27.639008 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:27.638968 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-776a4eb3-wxrfv_1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1/revoke-keys/0.log" Oct 31 01:34:28.309743 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:28.309713 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2cfd12b-jsbb9_99739e84-48b2-485e-9cc3-4eca5e4fb179/revoke-keys/0.log" Oct 31 01:34:30.580045 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:30.580007 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f7f7dc8cd-g96sp_91127689-97c8-4ca7-82a1-6d2d22f3a9c4/authorino/0.log" Oct 31 01:34:30.986628 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:30.986592 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqmtl_f877144e-05ba-4ead-a809-5daea44d1075/manager/0.log" Oct 31 01:34:31.231749 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:31.231715 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zp68g_d98fb307-575b-4e36-8e52-9398d884733a/manager/0.log" Oct 31 01:34:32.584310 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:32.584276 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6b4pn_8c5a0c2c-a45a-45d0-9e71-505ad5dff953/manager/0.log" Oct 31 01:34:33.026836 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:33.026801 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6k4fb_1fd906c8-7686-4cb4-9057-0d8b1da239d1/manager/0.log" Oct 31 01:34:33.863607 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:33.863572 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fgwg4f_bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699/istio-proxy/0.log" Oct 31 01:34:35.172629 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:35.172601 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj_2bbea852-9190-4206-a458-0ef7f9aa996f/istio-proxy/0.log" Oct 31 01:34:35.908226 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:35.908195 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cfd77cbdf-kcvgr_a8643bdf-eeca-4201-8d47-e4b1b69eefa2/router/0.log" Oct 31 01:34:37.017431 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:34:37.017394 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:34:37.420313 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:37.420236 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qpkjl_b7848d63-5555-4ae9-8ba1-934e520c43f7/main/0.log" Oct 31 01:34:38.189479 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:38.189450 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-mfrb7_ffdbfdaf-a6ca-4997-9cfd-acc37e1915b4/main/0.log" Oct 31 01:34:38.753838 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:38.753749 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-42ch2_b661f84e-8c0e-484f-a531-afc81020e2f5/main/0.log" Oct 31 01:34:49.324304 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324265 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dczd8/must-gather-598m7"] Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324633 5890 cpu_manager.go:414] "containerMap: removing container" podUID="2a114a39-0af2-437e-8b9e-1353ae7b7267" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324644 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a114a39-0af2-437e-8b9e-1353ae7b7267" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324660 5890 cpu_manager.go:414] "containerMap: removing container" podUID="a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324665 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324675 5890 cpu_manager.go:414] "containerMap: removing container" podUID="9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324680 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324688 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324693 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324703 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8a6b1212-8008-4f0e-a86f-638da400aeea" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324708 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6b1212-8008-4f0e-a86f-638da400aeea" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324717 5890 cpu_manager.go:414] "containerMap: removing container" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324723 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324730 5890 cpu_manager.go:414] "containerMap: removing container" podUID="8c4268ea-a56f-424c-b547-0bf580665e33" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324734 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4268ea-a56f-424c-b547-0bf580665e33" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324743 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324747 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324758 5890 cpu_manager.go:414] "containerMap: removing container" podUID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324763 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324769 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b6b0b7a9-4e13-42a7-809f-360661bb960e" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324773 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b0b7a9-4e13-42a7-809f-360661bb960e" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324779 5890 cpu_manager.go:414] "containerMap: removing container" podUID="4899c6c7-5566-4f4c-adb6-9e6c696d4838" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324796 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4899c6c7-5566-4f4c-adb6-9e6c696d4838" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324801 5890 cpu_manager.go:414] "containerMap: removing container" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324805 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324812 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e1879df4-734d-4002-9fff-4b5c1c1128e7" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324817 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1879df4-734d-4002-9fff-4b5c1c1128e7" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324825 5890 cpu_manager.go:414] "containerMap: removing container" podUID="dc16ba85-0f91-4224-8257-2b27b920a7d3" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324831 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc16ba85-0f91-4224-8257-2b27b920a7d3" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324838 5890 cpu_manager.go:414] "containerMap: removing container" podUID="2498dc56-4e7a-45f0-bc55-ce80a12e774b" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324842 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2498dc56-4e7a-45f0-bc55-ce80a12e774b" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324848 5890 cpu_manager.go:414] "containerMap: removing container" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324852 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324859 5890 cpu_manager.go:414] "containerMap: removing container" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324864 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324871 5890 cpu_manager.go:414] "containerMap: removing container" podUID="636e6616-e2a9-4113-8c62-11799d3ea892" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324875 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636e6616-e2a9-4113-8c62-11799d3ea892" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324881 5890 cpu_manager.go:414] "containerMap: removing container" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324885 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324894 5890 cpu_manager.go:414] "containerMap: removing container" podUID="71d79d95-9651-4aed-a942-e4d087b335d5" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324898 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d79d95-9651-4aed-a942-e4d087b335d5" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324904 5890 cpu_manager.go:414] "containerMap: removing container" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324909 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324921 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324925 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324932 5890 cpu_manager.go:414] "containerMap: removing container" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324937 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324943 5890 cpu_manager.go:414] "containerMap: removing container" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324947 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324954 5890 cpu_manager.go:414] "containerMap: removing container" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324959 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324967 5890 cpu_manager.go:414] "containerMap: removing container" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324972 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324978 5890 cpu_manager.go:414] "containerMap: removing container" podUID="c91f41c9-0702-4750-94d7-c660db68c749" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.324982 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91f41c9-0702-4750-94d7-c660db68c749" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325038 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4e2839-34eb-4d91-aaa2-969f5dea68d8" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325046 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b0b7a9-4e13-42a7-809f-360661bb960e" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325051 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="4899c6c7-5566-4f4c-adb6-9e6c696d4838" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325055 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325063 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1879df4-734d-4002-9fff-4b5c1c1128e7" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325069 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc16ba85-0f91-4224-8257-2b27b920a7d3" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325076 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="2498dc56-4e7a-45f0-bc55-ce80a12e774b" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325080 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325085 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325089 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="636e6616-e2a9-4113-8c62-11799d3ea892" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325094 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f176e3c-5114-48dd-880f-2150d4bc5b22" containerName="main" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325100 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d79d95-9651-4aed-a942-e4d087b335d5" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325105 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325110 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325115 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3815179-7ff3-4db0-b671-ecdafd4e977c" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325120 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325125 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="42be952b-15bf-41c7-b416-3f259d97849b" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325131 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325137 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91f41c9-0702-4750-94d7-c660db68c749" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325142 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a114a39-0af2-437e-8b9e-1353ae7b7267" containerName="payload-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325148 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f4edaf-4c2c-4345-9a20-61c27b2fc9af" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325154 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b545ee4-8ec4-4f82-8e1e-182d95a48bc1" containerName="maas-api" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325159 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00e0b1d-36be-4089-96a1-f35924e0fda9" containerName="istio-proxy" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325165 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6b1212-8008-4f0e-a86f-638da400aeea" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325170 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab22885-f796-4636-9e98-7622e5a6757a" containerName="main" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325175 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4268ea-a56f-424c-b547-0bf580665e33" containerName="payload-pre-processing" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325180 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="e078e70d-eeea-437a-994a-0a8a0c68f831" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325256 5890 cpu_manager.go:414] "containerMap: removing container" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325261 5890 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.326684 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.325311 5890 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5ceaed-5e34-4a0e-bf28-429fb7bfbce2" containerName="cleanup" Oct 31 01:34:49.329808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.328388 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.378454 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.378426 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dczd8\"/\"kube-root-ca.crt\"" Oct 31 01:34:49.378615 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.378532 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dczd8\"/\"openshift-service-ca.crt\"" Oct 31 01:34:49.396933 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.396898 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c47314c-e4a7-4d03-9bc2-6865d43a3913-must-gather-output\") pod \"must-gather-598m7\" (UID: \"9c47314c-e4a7-4d03-9bc2-6865d43a3913\") " pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.397064 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.396944 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snfm5\" (UniqueName: \"kubernetes.io/projected/9c47314c-e4a7-4d03-9bc2-6865d43a3913-kube-api-access-snfm5\") pod \"must-gather-598m7\" (UID: \"9c47314c-e4a7-4d03-9bc2-6865d43a3913\") " pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.425987 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.425956 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dczd8\"/\"default-dockercfg-b7mvt\"" Oct 31 01:34:49.427895 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.427871 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dczd8/must-gather-598m7"] Oct 31 01:34:49.429482 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.429461 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pq2gw/must-gather-jf2kv"] Oct 31 01:34:49.433475 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.433454 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:49.497477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.497450 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c47314c-e4a7-4d03-9bc2-6865d43a3913-must-gather-output\") pod \"must-gather-598m7\" (UID: \"9c47314c-e4a7-4d03-9bc2-6865d43a3913\") " pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.497650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.497491 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c527559-fe60-475a-8660-1f8ad0ee0f54-must-gather-output\") pod \"must-gather-jf2kv\" (UID: \"4c527559-fe60-475a-8660-1f8ad0ee0f54\") " pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:49.497650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.497514 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4cp\" (UniqueName: \"kubernetes.io/projected/4c527559-fe60-475a-8660-1f8ad0ee0f54-kube-api-access-sd4cp\") pod \"must-gather-jf2kv\" (UID: \"4c527559-fe60-475a-8660-1f8ad0ee0f54\") " pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:49.497650 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.497531 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snfm5\" (UniqueName: \"kubernetes.io/projected/9c47314c-e4a7-4d03-9bc2-6865d43a3913-kube-api-access-snfm5\") pod \"must-gather-598m7\" (UID: \"9c47314c-e4a7-4d03-9bc2-6865d43a3913\") " pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.497876 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.497775 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c47314c-e4a7-4d03-9bc2-6865d43a3913-must-gather-output\") pod \"must-gather-598m7\" (UID: \"9c47314c-e4a7-4d03-9bc2-6865d43a3913\") " pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.501815 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.501793 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pq2gw/must-gather-jf2kv"] Oct 31 01:34:49.559351 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.559317 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snfm5\" (UniqueName: \"kubernetes.io/projected/9c47314c-e4a7-4d03-9bc2-6865d43a3913-kube-api-access-snfm5\") pod \"must-gather-598m7\" (UID: \"9c47314c-e4a7-4d03-9bc2-6865d43a3913\") " pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.565934 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.565904 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pq2gw\"/\"openshift-service-ca.crt\"" Oct 31 01:34:49.566074 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.565978 5890 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pq2gw\"/\"kube-root-ca.crt\"" Oct 31 01:34:49.579333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.579283 5890 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pq2gw\"/\"default-dockercfg-4w94n\"" Oct 31 01:34:49.598728 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.598701 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c527559-fe60-475a-8660-1f8ad0ee0f54-must-gather-output\") pod \"must-gather-jf2kv\" (UID: \"4c527559-fe60-475a-8660-1f8ad0ee0f54\") " pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:49.598859 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.598733 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4cp\" (UniqueName: \"kubernetes.io/projected/4c527559-fe60-475a-8660-1f8ad0ee0f54-kube-api-access-sd4cp\") pod \"must-gather-jf2kv\" (UID: \"4c527559-fe60-475a-8660-1f8ad0ee0f54\") " pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:49.599052 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.599034 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c527559-fe60-475a-8660-1f8ad0ee0f54-must-gather-output\") pod \"must-gather-jf2kv\" (UID: \"4c527559-fe60-475a-8660-1f8ad0ee0f54\") " pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:49.642601 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.642573 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dczd8/must-gather-598m7" Oct 31 01:34:49.657187 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.657163 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4cp\" (UniqueName: \"kubernetes.io/projected/4c527559-fe60-475a-8660-1f8ad0ee0f54-kube-api-access-sd4cp\") pod \"must-gather-jf2kv\" (UID: \"4c527559-fe60-475a-8660-1f8ad0ee0f54\") " pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:49.751616 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:49.751585 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pq2gw/must-gather-jf2kv" Oct 31 01:34:50.068951 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:50.068920 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dczd8/must-gather-598m7"] Oct 31 01:34:50.071309 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:34:50.071282 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c47314c_e4a7_4d03_9bc2_6865d43a3913.slice/crio-1555f7f8ffc50fc094851924071ee4c5215f234ca90ce076234870c378f2d22e WatchSource:0}: Error finding container 1555f7f8ffc50fc094851924071ee4c5215f234ca90ce076234870c378f2d22e: Status 404 returned error can't find the container with id 1555f7f8ffc50fc094851924071ee4c5215f234ca90ce076234870c378f2d22e Oct 31 01:34:50.073200 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:50.073183 5890 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:34:50.136842 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:50.136815 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pq2gw/must-gather-jf2kv"] Oct 31 01:34:50.138108 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:34:50.138077 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c527559_fe60_475a_8660_1f8ad0ee0f54.slice/crio-91e103fe8080526d8e6bd57a29cdd905a3cb2f379c95e6dfc56d7cc7a5f0549f WatchSource:0}: Error finding container 91e103fe8080526d8e6bd57a29cdd905a3cb2f379c95e6dfc56d7cc7a5f0549f: Status 404 returned error can't find the container with id 91e103fe8080526d8e6bd57a29cdd905a3cb2f379c95e6dfc56d7cc7a5f0549f Oct 31 01:34:50.705647 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:50.705609 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dczd8/must-gather-598m7" event={"ID":"9c47314c-e4a7-4d03-9bc2-6865d43a3913","Type":"ContainerStarted","Data":"1555f7f8ffc50fc094851924071ee4c5215f234ca90ce076234870c378f2d22e"} Oct 31 01:34:50.706803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:50.706759 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pq2gw/must-gather-jf2kv" event={"ID":"4c527559-fe60-475a-8660-1f8ad0ee0f54","Type":"ContainerStarted","Data":"91e103fe8080526d8e6bd57a29cdd905a3cb2f379c95e6dfc56d7cc7a5f0549f"} Oct 31 01:34:52.722848 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:52.722806 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dczd8/must-gather-598m7" event={"ID":"9c47314c-e4a7-4d03-9bc2-6865d43a3913","Type":"ContainerStarted","Data":"504370e3b7ff877ea3f8253c995df47ee1d1c3b65f2f622cfdbfd1910f2898fe"} Oct 31 01:34:52.723334 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:52.722858 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dczd8/must-gather-598m7" event={"ID":"9c47314c-e4a7-4d03-9bc2-6865d43a3913","Type":"ContainerStarted","Data":"3463a778dd424a9e975bd359e6e90ac8b3d2f733a51f46727781a4ca85d542dc"} Oct 31 01:34:52.835560 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:52.835484 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dczd8/must-gather-598m7" podStartSLOduration=2.228604879 podStartE2EDuration="3.835467397s" totalImagesPullingTime="1.606862518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:52.834342682 +0000 UTC m=+3985.036593116" watchObservedRunningTime="2026-10-31 01:34:52.835467397 +0000 UTC m=+3985.037717829" Oct 31 01:34:59.771008 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:59.770973 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pq2gw/must-gather-jf2kv" event={"ID":"4c527559-fe60-475a-8660-1f8ad0ee0f54","Type":"ContainerStarted","Data":"f04fd8ec25ed54b7d40f9e079787196ba0119ab3becc33333343fab2a7c1eeac"} Oct 31 01:34:59.771435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:59.771017 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pq2gw/must-gather-jf2kv" event={"ID":"4c527559-fe60-475a-8660-1f8ad0ee0f54","Type":"ContainerStarted","Data":"6fd821b47ccf09d683a57ab9a8fbe27d4d3dc9d8d54c09861bac3d3280586ef8"} Oct 31 01:34:59.882258 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:34:59.882201 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pq2gw/must-gather-jf2kv" podStartSLOduration=2.326281734 podStartE2EDuration="10.88218637s" totalImagesPullingTime="8.555904636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:59.881259318 +0000 UTC m=+3992.083509750" watchObservedRunningTime="2026-10-31 01:34:59.88218637 +0000 UTC m=+3992.084436800" Oct 31 01:35:01.762803 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:01.758839 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ksxrk_09934062-de9f-4e88-b25e-4f4a7947e4ce/global-pull-secret-syncer/0.log" Oct 31 01:35:02.369845 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:02.368646 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-24h4x_ae509a6d-ba84-42c0-9a51-0e7971cb9d68/connection-checker/0.log" Oct 31 01:35:02.820152 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:02.819926 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-n7zhr_a4090b54-5b6a-49f9-b092-d7c309f3d913/connection-checker/0.log" Oct 31 01:35:03.012941 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:03.012900 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:35:03.259837 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:03.259801 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-prfjr_07e5eb4b-7a3b-43d5-81f9-cda790ace0b2/connection-checker/0.log" Oct 31 01:35:04.614247 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:04.614100 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nt477_c63af1f1-ad79-4694-955a-25eb509899d7/konnectivity-agent/0.log" Oct 31 01:35:05.802435 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:05.802394 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-105.us-west-2.compute.internal_d22eb601a1d03eddfb99de6ecc9e9dbd/haproxy/0.log" Oct 31 01:35:05.838364 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:05.838330 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:35:07.041284 ip-10-0-143-105 kubenswrapper[5890]: E1031 01:35:07.041246 5890 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jd4rk" containerName="extractor" Oct 31 01:35:08.462808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:08.461467 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:35:10.048114 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:10.048076 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv"] Oct 31 01:35:10.100877 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:10.100834 5890 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9"] Oct 31 01:35:10.101122 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:10.100899 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776a4eb3-wxrfv"] Oct 31 01:35:10.102477 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:10.102443 5890 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cfd12b-jsbb9"] Oct 31 01:35:11.533197 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.533159 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f7f7dc8cd-g96sp_91127689-97c8-4ca7-82a1-6d2d22f3a9c4/authorino/0.log" Oct 31 01:35:11.614891 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.614860 5890 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg"] Oct 31 01:35:11.622144 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.622120 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.677344 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.677306 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg"] Oct 31 01:35:11.715500 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.715453 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-proc\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.715924 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.715852 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7k67\" (UniqueName: \"kubernetes.io/projected/56d623df-a32f-488b-8b05-741faeae019f-kube-api-access-p7k67\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.716065 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.716008 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-sys\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.716142 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.716086 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-podres\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.716209 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.716143 5890 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-lib-modules\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.751427 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.751396 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:35:11.817333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817260 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-proc\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817333 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817295 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7k67\" (UniqueName: \"kubernetes.io/projected/56d623df-a32f-488b-8b05-741faeae019f-kube-api-access-p7k67\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817400 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-proc\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817455 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-sys\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817483 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-podres\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817555 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817519 5890 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-lib-modules\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817726 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817566 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-sys\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817726 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817587 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-podres\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.817726 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.817665 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d623df-a32f-488b-8b05-741faeae019f-lib-modules\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.865017 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.864980 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqmtl_f877144e-05ba-4ead-a809-5daea44d1075/manager/0.log" Oct 31 01:35:11.875713 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.875682 5890 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7k67\" (UniqueName: \"kubernetes.io/projected/56d623df-a32f-488b-8b05-741faeae019f-kube-api-access-p7k67\") pod \"perf-node-gather-daemonset-sq9fg\" (UID: \"56d623df-a32f-488b-8b05-741faeae019f\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:11.934677 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.934642 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1" path="/var/lib/kubelet/pods/1c5e5a70-e6ab-4b55-836c-23a5e0a04fc1/volumes" Oct 31 01:35:11.935112 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.935095 5890 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99739e84-48b2-485e-9cc3-4eca5e4fb179" path="/var/lib/kubelet/pods/99739e84-48b2-485e-9cc3-4eca5e4fb179/volumes" Oct 31 01:35:11.938855 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:11.938832 5890 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:12.294541 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:12.294507 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zp68g_d98fb307-575b-4e36-8e52-9398d884733a/manager/0.log" Oct 31 01:35:12.309990 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:12.309963 5890 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg"] Oct 31 01:35:12.312476 ip-10-0-143-105 kubenswrapper[5890]: W1031 01:35:12.312443 5890 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod56d623df_a32f_488b_8b05_741faeae019f.slice/crio-89e12cf363b48d63a2201b1b6a3d962e34050e52d12a896156c05affa334d5f2 WatchSource:0}: Error finding container 89e12cf363b48d63a2201b1b6a3d962e34050e52d12a896156c05affa334d5f2: Status 404 returned error can't find the container with id 89e12cf363b48d63a2201b1b6a3d962e34050e52d12a896156c05affa334d5f2 Oct 31 01:35:12.916444 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:12.916407 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" event={"ID":"56d623df-a32f-488b-8b05-741faeae019f","Type":"ContainerStarted","Data":"8cbb59e32258a0bafa3dbb48bfd2745d322699ef92279134dc3aa63abf49f1da"} Oct 31 01:35:12.916949 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:12.916472 5890 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" event={"ID":"56d623df-a32f-488b-8b05-741faeae019f","Type":"ContainerStarted","Data":"89e12cf363b48d63a2201b1b6a3d962e34050e52d12a896156c05affa334d5f2"} Oct 31 01:35:12.916949 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:12.916507 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:13.040068 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:13.040014 5890 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" podStartSLOduration=2.039999679 podStartE2EDuration="2.039999679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:13.038490419 +0000 UTC m=+4005.240740852" watchObservedRunningTime="2026-10-31 01:35:13.039999679 +0000 UTC m=+4005.242250102" Oct 31 01:35:13.580665 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:13.580618 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:35:13.904301 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:13.904217 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6b4pn_8c5a0c2c-a45a-45d0-9e71-505ad5dff953/manager/0.log" Oct 31 01:35:14.718027 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:14.717990 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6k4fb_1fd906c8-7686-4cb4-9057-0d8b1da239d1/manager/0.log" Oct 31 01:35:16.863145 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:16.863110 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-h677g_70dd7557-9f36-490f-8786-332d2a64de50/server/0.log" Oct 31 01:35:17.859755 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:17.859723 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fgwg4f_bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699/istio-proxy/0.log" Oct 31 01:35:18.934419 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:18.934391 5890 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-sq9fg" Oct 31 01:35:19.221886 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:19.221752 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj_2bbea852-9190-4206-a458-0ef7f9aa996f/istio-proxy/0.log" Oct 31 01:35:19.859049 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:19.859020 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cfd77cbdf-kcvgr_a8643bdf-eeca-4201-8d47-e4b1b69eefa2/router/0.log" Oct 31 01:35:20.297400 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:20.297364 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-xlk72_f4428250-46e8-4f67-bb0c-285938ee2175/cluster-monitoring-operator/0.log" Oct 31 01:35:21.050748 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:21.050716 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fgwg4f_bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699/istio-proxy/0.log" Oct 31 01:35:22.366303 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:22.366258 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pqzdj_2bbea852-9190-4206-a458-0ef7f9aa996f/istio-proxy/0.log" Oct 31 01:35:22.789439 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:22.789406 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wfdm_be133361-4e22-4617-92fd-610a57bef9e2/node-exporter/0.log" Oct 31 01:35:22.985885 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:22.985849 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5cfd77cbdf-kcvgr_a8643bdf-eeca-4201-8d47-e4b1b69eefa2/router/0.log" Oct 31 01:35:23.213480 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:23.213452 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wfdm_be133361-4e22-4617-92fd-610a57bef9e2/kube-rbac-proxy/0.log" Oct 31 01:35:23.645385 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:23.645316 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wfdm_be133361-4e22-4617-92fd-610a57bef9e2/init-textfile/0.log" Oct 31 01:35:25.395326 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:25.395291 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f7f7dc8cd-g96sp_91127689-97c8-4ca7-82a1-6d2d22f3a9c4/authorino/0.log" Oct 31 01:35:25.507575 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:25.507545 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqmtl_f877144e-05ba-4ead-a809-5daea44d1075/manager/0.log" Oct 31 01:35:25.711709 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:25.711672 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zp68g_d98fb307-575b-4e36-8e52-9398d884733a/manager/0.log" Oct 31 01:35:26.848051 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:26.848014 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6b4pn_8c5a0c2c-a45a-45d0-9e71-505ad5dff953/manager/0.log" Oct 31 01:35:27.242808 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:27.242755 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6k4fb_1fd906c8-7686-4cb4-9057-0d8b1da239d1/manager/0.log" Oct 31 01:35:29.028299 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:29.028265 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f7f7dc8cd-g96sp_91127689-97c8-4ca7-82a1-6d2d22f3a9c4/authorino/0.log" Oct 31 01:35:29.144890 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:29.144848 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqmtl_f877144e-05ba-4ead-a809-5daea44d1075/manager/0.log" Oct 31 01:35:29.357326 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:29.357236 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zp68g_d98fb307-575b-4e36-8e52-9398d884733a/manager/0.log" Oct 31 01:35:30.480681 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:30.480642 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6b4pn_8c5a0c2c-a45a-45d0-9e71-505ad5dff953/manager/0.log" Oct 31 01:35:30.876841 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:30.876763 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6k4fb_1fd906c8-7686-4cb4-9057-0d8b1da239d1/manager/0.log" Oct 31 01:35:35.250499 ip-10-0-143-105 kubenswrapper[5890]: I1031 01:35:35.250466 5890 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fgwg4f_bdd9901e-a5e0-47e3-8fb9-2bbfbfc05699/istio-proxy/0.log"